Are Survey Nonrespondents Willing to Provide Consent to Use Administrative

Records? Evidence from a Nonresponse Follow-Up Survey in Germany

Joseph W. Sakshaug (corresponding author)

School of Social Sciences

University of Manchester

Humanities Bridgeford Street-G12

Manchester M13 9PL

United Kingdom

Phone: +44 161 275 0271

Fax: +44 161 275 4722

Email: joe.sakshaug@manchester.ac.uk

Stephanie Eckman

RTI International

701 13th Street NW, Suite 750

Washington, DC 20005-3967

USA

Phone: +1 202 974 7816

Email: seckman@rti.org

Tables: 3

Figures: 1

References: 60

Word count: 6,547

Are Survey Nonrespondents Willing to Provide Consent to Use Administrative

Records? Evidence from a Nonresponse Follow-Up Survey in Germany

ABSTRACT

To mitigate the effects of low survey participation rates and possible nonresponse bias in

survey estimates, survey organizations often try to collect auxiliary information with which to

evaluate and possibly adjust for differences between respondents and nonrespondents. Call

record data and other forms of paradata are commonly used for this purpose, but these data

tend to be only weakly correlated with the survey items. Follow-up surveys conducted with

nonrespondents try to get around this issue by asking a subset of key items selected from the

original questionnaire. However, intensive follow-up procedures are expensive and have

other known limitations. In this article, we explore an alternative follow-up procedure which

simply asks nonrespondents for consent to use their administrative records in lieu of taking

part in a telephone survey interview. Utilizing a unique study design with administrative

records available for the overall study sample, we examine characteristics of telephone

nonrespondents who consent to record use in a mail follow-up survey. Interestingly, we find

that many telephone nonrespondents are willing to grant access to their administrative

records. These consenting nonrespondents are similar to the remaining survey

nonrespondents, yet different from the telephone survey respondents, which results in

reduced nonresponse bias for some key economic items. We discuss the practical

implications of these findings and offer some suggestions for incorporating the collected

administrative data in nonresponse bias evaluation and adjustment procedures.

Keywords: survey nonresponse, nonresponse bias, administrative records, consent

INTRODUCTION

Survey nonresponse is a pervasive problem that can reduce the quality of statistical estimates

as well as the ability to generalize those estimates to the target population. Nonresponse rates

have been increasing worldwide (Curtin, Presser, and Singer 2005; de Leeuw and de Heer

2002), raising serious concerns over whether surveys can still: 1) produce information that

meets high-quality standards demanded by various stakeholders, and therefore; 2) justify the

enormous amount of financial resources spent on their development and implementation. The

presence (or absence) of nonresponse bias in specific survey estimates is paramount in this

discussion. If increasing nonresponse rates coincide with increasing nonresponse bias for key

survey estimates, then the above concerns have merit. On the other hand, if increasing

nonresponse rates do not coincide with increasing nonresponse bias (or other sources of bias),

then concerns over the future of survey research may be overstated. Unfortunately, the only

way to address this issue and assess the extent of nonresponse bias for a given survey is to

possess at least some information about the nonrespondents.

In practice usually very little information is available for the nonrespondents. As a result,

assessing the existence and extent of nonresponse bias (and possibly adjusting for it) can be a

challenging exercise. The information that is available on nonrespondents usually comes

from a few different sources. For instance, studies that draw their samples from population

registers or special population lists (e.g., members of a specific health insurance provider)

often have access to rich auxiliary information for both respondents and nonrespondents.

However, beyond these unique situations, most general population surveys must rely on other

sources of auxiliary information, such as call records, doorstep interactions, interviewer

observations and other forms of paradata (Couper 1998; Couper and Lyberg 2005; Kreuter,

2013; Lynn 2003; Olson 2013a; Smith 2011; Walsh, Dahlhamer, and Bates 2013).

Sometimes additional information from public records, commercial databases, and aggregate

data can be linked to the sampled unit to supplement the collected paradata (e.g., Smith and

Kim 2013). But whether these data sources are effective for nonresponse bias assessment and

adjustment is questionable. For instance, while many studies find that paradata and other

auxiliary sources have strong correlations with the propensity to respond, they tend to be less

correlated with the collected survey variables with limited utility in nonresponse adjustment

(Biemer and Peytchev 2012, 2013; Kreuter and Kohler 2009; Kreuter et al. 2010; Lin and

Schaeffer 1995; Peytchev and Olson 2007; Sakshaug and Kreuter 2011) – an undesirable

property when used for reducing nonresponse bias in the collected survey variables (Little

and Vartivarian 2005).

To investigate the extent of nonresponse bias for specific survey variables, some studies

implement follow-up surveys with nonrespondents, which attempt to elicit information from

all (or a subsample of) remaining nonrespondents (Cobben and Schouten 2007; Couper et al.

2007; Olson, Lepkowski, and Garabrant 2011; Schouten 2007; Stoop et al. 2010; Voogt and

Van Kemper 2002). Intensive recruitment methods incorporating a variety of design features

(e.g., additional incentives, shortened questionnaire, switching data collection mode) are

often used to entice participation. Although nonresponse follow-up surveys virtually never

achieve complete response, they do succeed in converting a considerable portion of

nonrespondents and, in some cases, lead to reductions in nonresponse error, particularly when

different design protocols are used in the follow-up survey as compared to the initial survey

(Peytchev, Baxter, and Carley-Baxter 2009). However, despite these benefits, nonresponse

follow-up (NRFU) surveys have limitations which can restrict their usefulness in practice.

For instance, there is no guarantee that the follow-up respondents will be representative of the

remaining nonrespondents on key variables. In fact, NRFU surveys can increase nonresponse

bias if the remaining nonrespondents are even more selective than the initial nonrespondents.

Additionally, NRFU surveys are limited in the amount of information they collect – usually

only a small number of key items are asked – to encourage participation and minimize

response burden. Moreover, responses in follow-up survey data may be more prone to

measurement error compared to the initially-collected responses, particularly for items in

which respondents must recall past events and place them in time – the recall task becomes

more difficult for those who respond late in the field period (for a review of this literature, see

Olson 2013b).

An unexplored method of collecting information from nonrespondents, which may

overcome some of the above limitations, is to ask them for permission to use their

administrative records. Utilizing administrative records in survey research is not new. Several

surveys, including the U.S. Health and Retirement Study, the U.S. Panel Study of Income

Dynamics, the Survey of Income and Program Participation, the English Longitudinal Study

of Ageing, the Survey of Health, Aging and Retirement in Europe, the U.K. Understanding

Society, link to various types of administrative databases, including social security and

healthcare billing records (among others), which are then made available to researchers to

study a multitude of substantive and methodological research topics (Freedman, McGonagle,

and Andreski 2014; Knies and Burton 2014; Korbmacher and Czaplicki 2013; Olson 1999;

Sala, Burton, and Knies 2012). In some surveys, sections of the questionnaire are omitted

entirely, and interview length shortened accordingly, when respondents agree to link their

administrative records. For example, Statistics Canada has had some success in offering

respondents the choice of answering a series of income items in the survey, or permitting the

retrieval of this information from income tax records (Michaud et al. 1995). However, it is

much less common to ask for administrative records in lieu of taking part in the entire

interview, which is the approach we explore in this article.

There are potential advantages to collecting administrative information on

nonrespondents. One advantage is that administrative records often contain key substantive

outcomes that are related to topics covered in the survey questionnaire (e.g., income,

employment spells, healthcare utilization, benefit receipt). Thus, by making use of

administrative data, there is an opportunity to better understand differences between

respondents and nonrespondents on the relevant survey items and, in turn, improve

nonresponse bias adjustments. Moreover, many correlates of attrition in longitudinal studies

(e.g., mobility, changes in employment, health status) are recorded in administrative records,

thereby creating an opportunity to follow and track changes in nonrespondents over time

(Lemay 2009; Sakshaug and Huber 2016). Further, given their often longitudinal nature,

administrative databases contain a relatively accurate timeline of past events and relevant life

course changes. This information, in combination with similar information collected from

survey respondents, could be used to assess important differences between respondents and

nonrespondents on event history patterns. Additionally, by accessing this information from

administrative records, the response task becomes considerably less burdensome for follow-

up respondents who may otherwise be asked to recall this information from memory. Even

for items that do not require extensive recall, simply requesting access to administrative

records may be perceived as a less time-consuming and less burdensome task than answering

survey questions.

However, it is unclear whether nonrespondents would be willing to grant such a sensitive

request. Even respondents, who have already expressed a willingness to participate in a

survey, are often reluctant to consent to administrative record use because of privacy or other

concerns (Sala, Knies, and Burton 2014). For this reason, most studies that request consent to

administrative records use interviewer-administered modes, where interviewers can directly

address any questions and concerns that respondents may have. Interviewer-administration

generally yields higher rates of consent to administrative record use than self-administered

modes: in a review of the empirical literature, Fulton (2012) observed a mean consent rate of

75 percent, 63 percent, and 49 percent for in-person, telephone, and mail studies. However,

using interviewer-administered modes to follow-up with nonrespondents is expensive and

self-administered modes, such as mail, may offer a cheaper and more practical alternative to

collecting administrative information from nonrespondents. Studies that collect consent to

administrative records by mail are rare and tend to be conducted on special populations with

small sample sizes and a health focus (McCarthy et al. 1999; Murdoch, Pietila, and Partin

2010; Ziegenfuss et al. 2012). One exception is the study by Stone, Noel, and Weir (2013),

who conducted a pilot study of older persons (between the ages of 67 and 70 years old) who

had previously taken part in a large nationally-representative longitudinal study, but had not

been interviewed since 1974. The mail questionnaire included a request for consent to link

their responses to administrative data from the Social Security Administration. This

procedure yielded an overall consent rate of 21.2 percent (or 612 out of 2,886 fielded cases).

Experimental features embedded in the study, including the offering of financial incentives

for survey participation and a prompting of remaining nonrespondents by telephone, each had

a positive effect on the linkage consent rate.

In all of the cited mail surveys, consent to administrative record use was requested among

the entire study sample and conditional on survey participation. Reluctance to consent to

administrative records is likely to be more severe among survey nonrespondents and perhaps

more so among those who refused to take part in the survey as opposed to those for whom

contact could not be established. Refusers may choose not to participate in a survey due to

privacy or other concerns and thus may wish to withhold all of their information, including

administrative records, from research use. Furthermore, cases that receive a

disproportionately high number of call attempts may also be signaling their reluctance to

participate in the survey, even without explicitly refusing, by not making themselves

available. Consent rates to use administrative records may therefore be lower for cases

requiring the most recruitment effort. If refusers and cases requiring the most recruitment

effort consent at substantially lower rates than other nonrespondents, then the proposed

nonresponse follow-up (NRFU) procedure may be disadvantageous for addressing

nonresponse bias. Studies have shown that survey estimates can be differentially affected by

refusals and noncontacts, suggesting that different mechanisms underlie their occurrence

(Lynn and Clarke 2002; Olson, 2006; Sakshaug, Yan, and Tourangeau 2010). Altogether, to

be useful in practice, administrative records collected through NRFU procedures would be

needed for both refusals and noncontacts, and across the range of call attempts, in order to

account for different nonresponse mechanisms and adjust for them accordingly. What is also

important is that the nonrespondents who consent to the use of administrative records are

similar to those who do not consent based on the relevant substantive variables. Otherwise,

the record data may be less useful for evaluating and adjusting for nonresponse bias.

To shed further light on these issues, we report on a mail follow-up survey of

nonrespondents to a telephone survey in Germany. The follow-up survey included a request

for consent to use administrative employment records from the Federal Employment Agency

in Nuremberg. We utilize administrative data available for the entire sample to compare the

characteristics of the telephone respondents and the nonrespondents who consent to

administrative record use in the follow-up survey. Specifically, we address the following

research questions:

1. What proportion of telephone survey nonrespondents are willing to provide consent to

use their administrative records in a mail NRFU survey? Do consent rates in the mail

NRFU survey differ by telephone survey call record outcomes (e.g., refusals,

noncontacts, number of call attempts)? In particular, are consent rates lower for

refusals and for cases requiring the most call attempts versus noncontacts and those

requiring fewer call attempts?

2. Are the NRFU survey respondents who give consent to administrative record use

different from the telephone respondents on key administrative items? Are the NRFU

survey respondents who consent to administrative record use representative of the

remaining NRFU survey nonrespondents?

3. Does bringing in mail NRFU survey respondents who consent to administrative

record use and combining them with the telephone survey respondents reduce

nonresponse bias in the overall study sample for estimates derived from

administrative variables?

DATA AND METHODS

TELEPHONE SURVEY DATA COLLECTION

The computer-assisted telephone interviewing (CATI) survey was primarily conducted for

methodological purposes. The survey was fielded from August to October 2011 and consisted

of a 20-minute interview on topics related to employment activities and consumer behavior,

and contained several experiments relating to the order and format of filter and follow-up

questions, and the placement and wording of the question requesting respondent consent to

link their responses back to data contained in the administrative databases (Eckman et al.

2014; Sakshaug, Tutz, and Kreuter 2013). The overall study sample for the CATI survey

consisted of 12,400 named adults drawn from administrative databases maintained by the

Institute for Employment Research of the Federal Employment Agency in Germany (IAB

2011). The sample was drawn with equal probability in three mutually exclusive strata. The

first stratum consisted of persons who received income assistance in 2010 and were

employed in the past 10 years in a position that required social security contributions. The

second stratum was defined as persons who received unemployment benefits in the past 10

years, were employed in the past 10 years in a position requiring social security contributions,

and never received income assistance. The third stratum contained persons who had not

received income assistance nor unemployment benefits, and were employed with two or more

different employers requiring social security contributions in the past 10 years. More details

on the design and the reasoning behind the design are available in Eckman et al. (2014). A

total of 2,400 CATI survey respondents completed the interview for a response rate of 19.4

percent (AAPOR RR1; AAPOR 2016).

MAIL NONRESPONSE FOLLOW-UP SURVEY

After the close of the CATI survey, the nonresponse follow-up (NRFU) survey began. The

NRFU survey consisted of a mail questionnaire sent to CATI nonrespondents. Exclusions

included those who had radically refused the CATI survey (n=480),1 those for whom the

mailing address on the frame was not up to date (n=1,358), and those who were deceased

(n=40) or unable to participate due to a language barrier (n=242) or for health reasons (n=37).

This resulted in an eligible sample size of 7,843 persons. A random half of these cases

(n=3,922) was allocated to the NRFU sample, while the other half (n=3,921) was assigned to

a separate survey in which consent to access administrative records was not requested; the

results of the latter study are not reported here. The first mailing took place on December 6,

2011 and all remaining nonrespondents were mailed again on January 11, 2012. Each mailing

included a postage-paid, addressed envelope to facilitate return of the form.

The NRFU questionnaire consisted of two background questions (year of birth and

household size) and asked for consent to access administrative employment records. The

questionnaire was on one side of one page and was accompanied by a letter written by the

1 Radical (or hard) refusals were ones where the household or target person refused so strenuously that we felt it was better not to contact them again. Less radical (or soft) refusals were eligible for the NRFU.

study’s principal investigator explaining the rationale for the follow-up survey. The relevant

parts of the NRFU letter and questionnaire are presented below. The letter contained the

following excerpts, which are relevant to the present study (ENGLISH TRANSLATION):

“[…] Unfortunately, it was not possible to complete a full interview with you. We

understand that many people are busy and don’t always have time to take part in

studies like this. However, the labor market research we conduct is important all over

Germany. To facilitate participation, we have attached a shorter questionnaire to this

letter, which contains only a few questions from our study. […] There is also the

opportunity to gather material for our study and at the same time reduce the burden

for people just like you -- by taking advantage of data that is already available from

the authorities. The Federal Employment Agency has information about employment

and unemployment patterns in Germany. With your permission, we could use these

data for our study and combine them with the survey data. For this reason, we ask you

to give us consent to use these data for our investigation. We therefore kindly ask you

to return the completed questionnaire with the enclosed stamped envelope. […]”

and the questionnaire itself contained the following declaration of consent:

“I hereby agree that the Institute for Employment Research may use data about my

employment and unemployment patterns maintained by the Federal Employment

Agency for the purpose of the ‘Work and Consumer Behavior in Germany’ study.”

A space was provided to collect the respondent’s signature. In Germany, a signature is

required in order to make a consent declaration valid. It was possible to return the

questionnaire without giving consent, and 38 respondents did so.

ADMINISTRATIVE DATA

The administrative database from which the CATI sample was drawn contains extensive

information concerning employment history, social security contributions, and administrative

services provided by the Federal Employment Agency. The database contains detailed

information about employment and unemployment durations (or spells), benefit receipt, job

seeking and training programs, and earnings for all persons who have contributed to the

social security system from 1975 for West Germany and from 1992 for East Germany

(Bender and Haas 2002; Jacobebbinghaus and Seth 2007).

We use these data to address research questions 2 and 3: specifically, we analyze

characteristics of respondents who give consent in the mail NRFU survey relative to the

CATI respondents and remaining nonrespondents. We make use of the following

characteristics in the administrative data: age (in years; recoded: 28 or younger, 29-41, 42-54,

and 55 or older); sex; foreign nationality; education level (secondary, university entrance

qualification, college/university, and a missing data category2); ever employed part-time; ever

employed full-time; monthly earnings (in euros) from last full-time job (recoded: 0-600, 601-

1,700, 1,701-2,800, and 2,801 or more); earnings from last part-time job (recoded: 0-300,

301-700, 701-1,400, and 1,401 or more); receipt of unemployment benefit in 2010 and;

receipt of income assistance in 2010. The age and monthly earnings variables are categorized

according to their distributions to prevent unusually sparse categories, but are otherwise

formed rather arbitrarily. Continuous versions of these variables are also used and their

means are analyzed in the Results section. All of these characteristics are measured on the

date the administrative data were extracted for the study (June 30th, 2011). A correlation

analysis revealed modest levels of overlap between some variables; for example, the 2 The education variable in the administrative data suffers from a high rate of missing data; in our sample, education was missing for about 34 percent of cases. Strategies have been developed for imputing this variable at the Federal Employment Agency (Fitzenberger, Osikominu, and Völter 2005), but these strategies are not applied in the present study.

correlation between monthly earnings from last full-time job and income assistance in 2010,

and between monthly earnings from last full-time job and education (college/university) was

-0.39 and 0.37, respectively. However, the majority of correlations were much smaller and

therefore no data reduction procedures were considered. The full correlation matrix can be

found in Appendix Table 1.

The primary rationale for using these administrative variables in our analysis of the NRFU

survey is that they are commonly utilized in labor market research studies and are key

measures used by the Federal Employment Agency to administer social benefits. The

importance of these variables in other survey methodological work has also been

demonstrated (Kirchner 2015; Kreuter, Müller, and Trappmann 2010; West, Kreuter, and

Jaenichen 2013). Moreover, we chose these variables because they are very similar to items

that were collected in the main CATI survey, including past employment activities, earnings,

and receipt of unemployment benefit and income assistance. We believe this situation

simulates an ideal scenario for evaluating this NRFU procedure as any reductions in

nonresponse bias on the administrative estimates (due to the NRFU procedure) are likely to

translate to reductions in nonresponse bias for related survey items.

Because the overall study sample was selected from the administrative data, we already

had all the relevant economic information for both respondents and nonrespondents. That is,

in our special case, consent was not necessary to analyze the administrative data for

respondents and nonrespondents. According to data protection laws in Germany, consent

would be required to merge the administrative data with survey data, such as the two

background questions asked in the NRFU survey (Federal Data Protection Act 2013). For our

investigation, however, we do not link any survey responses from the NRFU or the CATI

survey to the administrative data.3 All procedures used in this study were approved by the 3 We do not analyze the responses to the two substantive questions in the NRFU survey, but other researchers following a similar nonresponse follow-up procedure would likely want to include questions about attitudes or

data protection legal team at the Institute for Employment Research. The fact that we are able

to analyze the administrative data for the entire sample makes our study quite unique and

allows us to do analyses that inform the use of administrative data by other researchers who

are not in our situation.

ANALYSIS PLAN

In describing the results we make use of explicit, mutually exclusive group titles and labels to

define each subgroup that is analyzed. We refer to the following group titles and labels

throughout the text: Overall Study Sample (Group A), CATI Respondents (Group B), CATI

Nonrespondents (Group C), NRFU Sample (Group D), all NRFU Respondents (Group E),

NRFU Nonrespondents (Group F), NRFU Respondents with Consent (Group G), and NRFU

Respondents without Consent (Group H). A flow chart depicting each stage of the study

along with group titles, labels, and sample sizes is depicted in Figure 1.

To address research questions 1 and 2, we report proportions (or means, for continuous

variables) with linearized standard errors, and use chi-squared test statistics to test for

differences between respondents and nonrespondents in both the CATI and NRFU surveys

based on the administrative variables. For research question 3, we report estimates of

percentage relative nonresponse bias and their standard errors to describe differences between

the respondents (before and after the NRFU survey) and the overall study sample (Groves

2006). Percentage relative nonresponse bias estimates are calculated as the difference

between the respondent-based estimate Y B (which is either a proportion or mean in our

setting) for the CATI respondents (Group B), or Y B+G for the combined CATI respondents

(Group B) and NRFU respondents with consent (Group G), and the overall study sample

behaviors that are not available in the administrative data.

(Group A) estimate Y A, which is then divided by the overall study sample estimate Y A and

multiplied by 100:

Percentage Relative Nonresponse Bias (Before NRFU Survey) = 100 ∙( Y B−Y A

Y A), and

Percentage Relative Nonresponse Bias (After NRFU Survey) = 100 ∙( Y B+G−Y A

Y A).

These estimates are intended to give an indication of the proportionate effect size of

nonresponse bias before and after the NRFU survey.

[INSERT FIGURE 1 ABOUT HERE]

Figure 1. Flow Chart with Group Titles and Labels for Each Relevant Study Group

RESULTS

OVERALL CONSENT RATE AND BREAKDOWN BY CATI SURVEY OUTCOME

The NRFU survey results are presented in Table 1; this table can be used to address the

first set of research questions. A total of 539 (out of 3,922) NRFU respondents (Group E), or

13.7 percent, returned the mail NRFU questionnaire and 501 of those cases (Group G)

consented to the use of administrative records, for an overall consent rate of 12.8 percent. The

last column of Table 1 shows that consent rates did not differ significantly by type of

nonresponse (refusal and noncontact) nor by the number of call attempts made to CATI

nonrespondents.

[INSERT TABLE 1 ABOUT HERE]

Because the purpose of the NRFU survey was to obtain consent to use administrative

records, in the remainder of the article we treat the 38 NRFU respondents who did not

provide consent (Group H) as nonrespondents in the NRFU survey and analyze them together

with the NRFU nonrespondents (Group F) in all subsequent analyses.

COMPOSITIONAL DIFFERENCES BETWEEN CATI RESPONDENTS, FOLLOW-UP

RESPONDENTS, AND REMAINING NONRESPONDENTS

Next, we examine characteristics of respondents and nonrespondents to the CATI and NRFU

surveys based on the administrative variables to address the second set of research questions.

The results are presented in Table 2; sample sizes for each administrative variable category

broken down by CATI and NRFU survey subgroups are shown in Appendix Table 2.

Looking at the CATI survey results first, we see there are significant differences between the

CATI respondents (Group B) and nonrespondents (Group C) for nearly all of the

administrative variables: CATI respondents are significantly more likely than their

nonresponding counterparts to be older, female, of German nationality, ever employed as

part-time and/or full-time, have lower full-time monthly earnings but higher part-time

monthly earnings, and have received unemployment benefit and income assistance in 2010

(see Table 2, Test 1). That is, the CATI nonrespondents are different from the respondents on

the administrative variables, indicating nonresponse bias in the CATI survey data.

[INSERT TABLE 2 ABOUT HERE]

Turning now to the mail NRFU survey results in Table 2, some similar and different

patterns emerge. As in the CATI survey, the NRFU respondents who gave consent (Group G)

are more likely to be older, female, of German nationality, and have higher part-time monthly

earnings compared to the combined nonrespondent group of NRFU nonrespondents (Group

F) and NRFU respondents without consent (Group H). However, in contrast to the findings in

the CATI survey, NRFU respondents who gave consent (Group G) have higher full-time

earnings and a lower prevalence of receiving income assistance in 2010 compared to the

combined NRFU nonrespondents (Groups F+H). Moreover, there are no significant

differences between the NRFU respondents with consent (Group G) and the combined NRFU

nonrespondents (Groups F+H) on education, full- and part-time employment status, full-time

monthly earnings (categorized), and unemployment benefit (see Table 2, Test 2), suggesting

that NRFU respondents who give consent are rather representative of the combined NRFU

nonrespondents (Groups F+H) on these characteristics, at least from a statistical testing

viewpoint.4 In short, the NRFU survey seems to have done a relatively good job of capturing

the characteristics of the CATI nonrespondents.

The last column of Table 2 (Test 3) tests whether respondents in the two surveys (CATI

and NRFU) are different from each other. We see that the NRFU respondents who give

consent (Group G) are significantly older, higher part-time (continuous) and full-time

earners, and less likely to receive unemployment benefit and income assistance in 2010

compared to all CATI respondents (Group B). In summary, the NRFU survey has made the

respondent pool more diverse, which suggests that it has reduced nonresponse bias in the

overall study sample: we examine the reduction in bias explicitly in the next step.

NONRESPONSE BIAS BEFORE AND AFTER NONRESPONSE FOLLOW-UP

The last set of analyses examines whether combining the CATI respondents (Group B) and

NRFU respondents who give consent (Group G) reduces nonresponse bias in the overall

4 The same finding holds if the 480 persons who “radically refused” the CATI survey are included in the combined NRFU nonrespondent pool (results not shown).

study sample (Group A) compared to the CATI only survey. This third and final research

question is the most important one for researchers who are interested in using the same

NRFU technique to reduce nonresponse bias.

Percentage relative nonresponse bias estimates are shown in Table 3 (bias estimates in

their original units can be found in Appendix Table 3). As the table shows, including the

NRFU respondents does not change the estimates dramatically – that is, the relative bias

estimates do not substantially change, and not all changes result in bias reduction: slight

increases in bias occur for age, sex, and part-time monthly earnings. Bias is reduced for

foreign citizenship, part-time and full-time employment, full-time monthly earnings,

unemployment benefit, and income assistance. Income assistance exhibits the largest relative

bias reduction, decreasing by more than 50 percent from a relative bias of 8.96 to 4.22

percent. Interestingly, the bias reduction for this item coincides with a reversal of the

statistically significant difference between CATI respondents (Group B) and CATI

nonrespondents (Group C) that was found in Table 2 (Test 1). That is, combining the NRFU

respondents who give consent (Group G) with all CATI respondents (Group B) yields a

respondent estimate of income assistance that is no longer statistically significantly different

to the estimate based on all remaining CATI nonrespondents (Groups C–G) (see Table 3, last

column).

A similar finding is observed for full-time monthly earnings: the smaller proportion of

high full-time earners in the CATI survey is offset by the larger proportion of high full-time

earners in the NRFU survey, which reduces the disparity between the combined respondents

(Groups B+G) and all remaining CATI nonrespondents (Groups C–G), resulting in a non-

significant difference between the two. Lastly, it is worth noting that the reduction in

nonresponse bias for these substantive estimates, namely, income assistance and full-time

monthly earnings, is observed despite increasing nonresponse bias for demographic estimates

(age and sex).

[INSERT TABLE 3 ABOUT HERE]

DISCUSSION

This study’s findings can be summarized into five main points. First, we found that about 13

percent of nonrespondents to a CATI survey that achieved a response rate of 19.4 percent

were willing to consent to the use of their administrative records in a mail nonresponse

follow-up survey. Second, consent to administrative record use did not differ by the type of

CATI nonresponse (noncontacts, refusals) nor by the number of call attempts. Third, while

the CATI respondents were not representative of the CATI nonrespondents on nearly all

administrative variables, the NRFU respondents who gave consent to administrative record

use were representative of the combined group of NRFU nonrespondents and NRFU

respondents who withheld consent on key substantive variables related to full-time earnings,

unemployment benefit receipt, and part- and full-time employment. A related fourth finding

is that the NRFU respondents who gave consent differed from the CATI respondents on

nearly all of the same economic variables mentioned in the previous sentence (and the

income assistance variable). Finally, combining all CATI respondents and the NRFU

respondents who gave consent reduced nonresponse bias in the overall study sample for

estimates of income assistance and full-time earnings, and further reduced the discrepancy

between these respondents and all remaining CATI nonrespondents to the point at which they

were no longer significantly different from each other for these two variables.

At this moment in time, when survey participation rates are at an all-time low and the

availability of individual-level auxiliary information beyond basic demographic details,

paradata, and linked commercial/public-use data is scarce, it is interesting to know that some

nonrespondents are willing to allow their administrative records to be used for research

purposes. Administrative records, which can contain a high-level of substantive information

about individuals, would seem to be a useful addition to existing auxiliary data sources being

used for nonresponse bias analysis. This statement is supported by the study’s findings that

the NRFU survey brought in respondents who consented to administrative record use and

who differed from respondents to the initial CATI survey, yet were relatively representative

of those who withheld consent either by not returning the consent form or returning the form

without a signature, which led to nonresponse bias reductions for some key substantive

estimates. Notably, these bias reductions were present despite increasing nonresponse bias for

some demographic estimates. A relevant practical implication here is that simply interpreting

changes in demographic variables, which are often the only individual-level information

available for nonrespondents, may lead to misleading conclusions concerning the

effectiveness of NRFU surveys for reducing nonresponse bias in other, more substantive,

variables.

The fact that we were able to analyze administrative data for the overall study sample and

validate the effectiveness of the nonresponse follow-up survey for reducing nonresponse bias

in key substantive variables, including those seemingly related to the variables collected in

the initial CATI survey, is an important and unique strength of this study. While most surveys

lack relevant auxiliary information to conclusively determine whether their NRFU procedures

are effective in reducing nonresponse bias, it is reassuring to know that such procedures,

including those that ask a relatively sensitive question about accessing administrative records,

can in fact improve the substantive composition of the overall study sample. Hence, it is our

hope that the results of this study can inform the use of administrative data for researchers

who are not in our situation.

An overarching question that arises from this study is: how can administrative data

collected from NRFU respondents be utilized for assessing and adjusting for nonresponse

bias? We see two possibilities here. The first possibility is to combine survey data for

respondents and administrative data for NRFU respondents. Survey organizations could

collect proxy variables from survey respondents; proxy in the sense that they closely

resemble key variables obtainable from the target administrative database. In our study, that

would have meant asking questions in the CATI survey about, for example, whether or not

the respondent had received some form of unemployment or supplemental income benefit in

2010, or an approximate (or bracketed) monthly income amount from their last full- or part-

time employment. Survey and administrative variables could then be merged to provide a

larger and hopefully more representative set of variables used for measuring and adjusting for

nonresponse bias. However, it may be that measurement and/or definitional differences exist

between the survey and administrative items, which could compromise the nonresponse

analysis. For example, a respondent reporting their income in a survey may be asked to

include income from additional sources (e.g., self-employment) that may not be captured in

the administrative record. If such measurement differences can be minimized, then

incorporating both sets of items in nonresponse procedures (e.g., weighting) becomes more

defensible.

A second possible use of administrative records collected from NRFU respondents is to

merge those records with administrative records collected from the main survey respondents,

and utilize both sets of records in nonresponse adjustment procedures. This approach

eliminates the issues with differential measurement in survey and administrative data

discussed above. A drawback is that administrative records are not made available for all

survey respondents because some respondents do not give consent to link their survey and

administrative records, or the administrative record in question does not exist or cannot be

linked for other reasons. As noted from the outset, many large-scale surveys ask respondents

for consent to link their survey responses with administrative records; in the present CATI

survey, about 95 percent of respondents consented to this linkage (Sakshaug et al. 2013).

However, it is worth noting that systematic differences often exist between respondents who

consent to linkage and respondents who do not, producing a consent bias (Al Baghal, Knies,

and Burton 2014; Bates 2005; Jenkins et al. 2006; Mostafa 2015; Sakshaug et al. 2012).

Moreover, linkage consent biases can behave differently than nonresponse biases, and do not

always shift estimates in the same direction (as shown in Sakshaug and Huber 2016;

Sakshaug and Kreuter 2012). Thus, a direct comparison between consenting respondents in

the main survey and the NRFU survey might lead to misleading conclusions regarding the

actual impact of nonresponse bias.

We acknowledge several factors that could limit the generalizability of the study’s

findings. First, while the study population includes the vast majority of people who work in

Germany, certain groups that do not contribute to the social security system were excluded,

such as civil servants (e.g., teachers, police officers), the self-employed, and long-term

retirees. Second, the study sponsor, the Federal Employment Agency, is a salient entity in

Germany, particularly for people who receive unemployment or income-related benefits,

which may have had a positive (or negative) influence on people’s decision to provide

consent to access their administrative employment records. It is unclear whether or not a

lesser-known sponsor, or a sponsor without direct ties to the relevant administrative database,

would have yielded a similar consent rate. A further unclear point is the extent to which the

change in data collection mode from CATI to mail influenced people’s willingness to

consent. While mode switches for NRFU surveys are relatively common we suspect that the

mail mode had a positive effect on the consent rate due to the visual features of the materials

(e.g., official government letterhead, personal letter written by the principal investigator),

postage-paid envelope, and enclosed data protection statement – all of which likely added to

the perceived credibility of the study; however, similar materials were also sent to the overall

study sample in an advance letter prior to the start of the CATI survey. Furthermore,

switching to the mail mode was undoubtedly helpful in reaching persons whose listed

telephone number was invalid.

We suspect that employing an in-person NRFU procedure would have yielded a higher

consent rate than the current mail procedure based on findings from different mode studies

(Fulton 2012), but likely at a much greater cost than was incurred here. Researchers must

weigh the cost-utility tradeoff of employing different modes to obtain administrative data

authorization. Disentangling the effect of mode on consent rates and bias and their associated

costs in NRFU surveys would be a natural topic for future exploration. A further point worth

exploring is the impact of requiring different types of information to document an

individual’s consent. The present study only required a signature by the respondent, but more

detailed information may be required to facilitate look-up of the relevant record. For this

reason, many studies accompany the request for administrative record consent with a request

for a unique identification number, such as a Social Security number, as was the case in the

Stone, Noel, and Weir (2013) study. However, asking for such a sensitive piece of identifying

information generally has a negative effect on the consent rate. For example, Sakshaug et al.

(2012) observed that 67.8 percent of respondents consented to the linkage of their survey

responses to Social Security records in the 2006 Health and Retirement Study, but only 49.1

percent of respondents were willing to consent and provide a Social Security number. Even

the provision of a signature can have a detrimental effect on consent, as shown by Sala et al.

(2014) who found that 3.9 percent of the Understanding Society sample who provided verbal

consent to administrative record use refused to sign a consent form. Research efforts are

needed to evaluate the effect of requiring such provisions in NRFU studies on the utility of

the collected records. Lastly, it is important to acknowledge that the present study was largely

a methodological exercise and at least some of the questions in the survey addressed matters

covered in the administrative records. In non-methodological surveys, the content of the

survey might include subject matters not available in records. In such surveys, record

information would only be useful for addressing nonresponse bias to the extent that the

administrative variables are correlated with the subject matter variables.

Notwithstanding these factors, this study finds that collecting permission to access

administrative records for some nonrespondents is feasible and may provide a cost-effective

alternative to more traditional and expensive NRFU efforts. There are some indications that

these data may be useful for assessing and adjusting for nonresponse bias, though we readily

acknowledge that identifying the optimal use of these data for this purpose requires additional

investigations. For this NRFU procedure to be useful in practice, many logistical questions

must be answered. Perhaps most prominent of all: Will administrative data custodians permit

the retrieval of, and access to, records for survey nonrespondents? For surveys that already

link their respondents’ survey answers to administrative databases this may be a more

straightforward proposition, particularly if similarly stringent consent procedures and data

protection safeguards are used for the nonrespondents. However, surveys often seek consent

from respondents for the purpose of linking their survey responses with administrative

records. If the administrative records are sought only for internal, methodological purposes

(e.g., nonresponse bias evaluation and/or adjustment) and are only linked to paradata and not

to the substantive survey responses, then consent may not be needed, as was the case in the

present study. Data privacy laws, which vary by country and local jurisdictions, and relevant

ethics committees would need to be consulted to determine the level and extent to which

consent may or may not be necessary.

REFERENCES

The American Association for Public Opinion Research (AAPOR). 2016. Standard

Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th

Edition. AAPOR.

Al Baghal, Tarek, Gundi Knies, and Jonathon Burton. 2014. “Linking Administrative

Records to Surveys: Differences in the Correlates to Consent Decisions.” Understanding

Society Technical Report, No. 2014-09, Institute for Social and Economic Research,

Essex, UK. < https://www.understandingsociety.ac.uk/research/publications/working-

paper/understanding-society/2014-09>

Bates, Nancy. 2005. “Development and Testing of Informed Consent Questions to Link

Survey Data with Administrative Records.” Paper Presented at the Joint Statistical

Meetings of the American Statistical Association, Minneapolis, MN, August. <

http://www.amstat.org/committees/ethics/linksdir/Jsm2005Bates.pdf>

Bender, Stefan, and Anette Haas. 2002. “Die IAB-Beschäftigtenstichprobe.” In G. Kleinhenz

(ed.) IAB-Kompendium Arbeitsmarkt- und Berufsforschung. Beiträge zur Arbeitsmarkt-

und Berufsforschung, BeitrAB 250: 3–12.

<http://doku.iab.de/beitrab/2002/beitr250_101.pdf>

Biemer, Paul P. and Andy Peytchev. 2012. “Census Geocoding for Nonresponse Bias

Evaluation in Telephone Surveys: An Assessment of the Error Properties.” Public

Opinion Quarterly 76:432–52.

Biemer, Paul P. and Andy Peytchev. 2013. “Using Geocoded Census Data for Nonresponse

Bias Correction: An Assessment.” Journal of Survey Statistics and Methodology 1:24–44.

Cobben, Fannie, and Barry Schouten. 2007. “A Follow-Up with Basic Questions of

Nonrespondents to the Dutch Labour Force Survey.” Discussion Paper 07011, Statistics

Netherlands. <http://www.cbs.nl/NR/rdonlyres/42B0DC4A-D071-46F8-A249-

408D7786E2F3/0/200711x10pub.pdf>

Couper, Mick P. 1998. “Measuring Survey Quality in a CASIC Environment.” Paper

presented at the Joint Statistical Meetings of the American Statistical Association, Dallas,

TX, August. < http://www.amstat.org/sections/srms/proceedings/papers/1998_006.pdf>

Couper, Mick P., and Lars E. Lyberg. 2005. “The Use of Paradata in Survey Research.”

Paper presented at the 54th Session of the International Statistical Institute, Sydney,

Australia, April.

Couper, Mick P., Andy Peytchev, Victor J. Strecher, Kendra Rothert, and Julia Anderson.

2007. “Following Up Nonrespondents to an Online Weight Management Intervention:

Randomized Trial Comparing Mail versus Telephone.” Journal of Medical Internet

Research 9(2):e16

Curtin, Richard, Stanley Presser, and Eleanor Singer. 2005. “Changes in Telephone Survey

Nonresponse over the Past Quarter Century.” Public Opinion Quarterly 69:87–98.

De Leeuw, Edith D. and Wim de Heer. 2002. “Trends in Household Survey Nonresponse: A

Longitudinal and International Comparison.” In Survey Nonresponse, eds. Robert M.

Groves, Don A. Dillman, John L. Eltinge, and Roderick J.A. Little, pp. 41–54. New York:

Wiley.

Eckman, Stephanie, Frauke Kreuter, Antje Kirchner, Annette Jäckle, Roger Tourangeau, and

Stanley Presser. 2014. “Assessing the Mechanisms of Misreporting to Filter Questions in

Surveys.” Public Opinion Quarterly 78:721–33.

Federal Data Protection Act. 2013. “Admissibility of Data Collection, Processing and Use.”

Bundesministerium der Justiz und für Verbraucherschutz. http://www.gesetze-im-

internet.de/englisch_bdsg/index.html

Fitzenberger, Bernd, Aderonke Osikominu, and Robert Völter. 2005. “Imputation Rules to

Improve the Education Variable in the IAB Employment Subsample.” Discussion

Paper 05-010, ZEW-Centre for European Economic Research. <

http://www.zew.de/de/publikationen/2489>

Freedman, Vicki A., Katherine McGonagle, and Patricia Andreski. 2014. “The Panel Study

of Income Dynamics Linked Medicare Claims Data.” PSID Technical Report Series 14-

01, University of Michigan.

Fulton, Jenna. 2012. “Respondent Consent to Use Administrative Data.” Doctoral

dissertation, University of Maryland.

Groves, Robert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.”

Public Opinion Quarterly 70:646–675.

Institut für Arbeitsmarkt- und Berufsforschung (IAB). 2011. Nuremberg, Germany:

Integrierte Erwerbsbiographien (IEB) V09.00.

Jacobebbinghaus, Peter, and Stefan Seth. 2007. “The German Integrated Employment

Biographies Sample IEBS.” Schmollers Jahrbuch 127:335–42.

Jenkins, Stephen P., Lorenzo Cappellari, Peter Lynn, Annette Jäckle, and Emanuela Sala.

2006. “Patterns of Consent: Evidence from a General Household Survey.” Journal of the

Royal Statistical Society: Series A (Statistics in Society) 169:701–22.

Kirchner, Antje. 2015. “Validating Sensitive Questions: A Comparison of Survey and

Register Data.” Journal of Official Statistics 31:31–59.

Knies, Gundi, and Jonathan Burton. 2014. “Analysis of Four Studies in a Comparative

Framework Reveals: Health Linkage Consent Rates on British Cohort Studies Higher

than on UK Household Panel Surveys.” BMC Medical Research Methodology 14:125.

Korbmacher, Julie, and Christin Czaplicki. 2013. “Linking SHARE Survey Data with

Administrative Records: First Experiences from SHARE-Germany. ” In SHARE Wave 4:

Innovations & Methodology, eds. Frederic Malter and Axel Börsch-Supan, pp. 47–

53.Munich: MEA, Max Planck Institute for Social Law and Social Policy.

Kreuter, Frauke. 2013. Improving Surveys with Paradata: Analytic Uses of Process

Information. Hoboken, New Jersey: Wiley.

Kreuter, Frauke, and Ulrich Kohler. 2009. “Analyzing Contact Sequences in Call Record

Data: Potential and Limitations of Sequence Indicators for Nonresponse Adjustments in

the European Social Survey.” Journal of Official Statistics 25:203–26.

Kreuter, Frauke, Gerrit Müller, and Mark Trappmann. 2010. “Nonresponse and Measurement

Error in Employment Research: Making Use of Administrative Data.” Public Opinion

Quarterly 74:880–906.

Kreuter, Frauke, Kristen Olson, James Wagner, Ting Yan, Trena M. Ezzati‐Rice, Carolina

Casas‐Cordero, Michael Lemay, Andy Peytchev, Robert M. Groves, and Trivellore E.

Raghunathan. 2010. “Using Proxy Measures and Other Correlates of Survey Outcomes to

Adjust for Non‐Response: Examples from Multiple Surveys.” Journal of the Royal

Statistical Society: Series A (Statistics in Society) 173:389–407.

Lemay, Michael. 2009. “Understanding the Mechanism of Panel Attrition.” Unpublished

Dissertation. University of Maryland. < http://drum.lib.umd.edu/handle/1903/9631>

Lin, I-Fen, and Nora Cate Schaeffer. 1995. “Using Survey Participants to Estimate the Impact

of Nonparticipation.” Public Opinion Quarterly 59:236–58.

Little, Roderick J.A., and Sonya Vartivarian. 2005. “Does Weighting for Nonresponse

Increase the Variance of Survey Means?” Survey Methodology 31:161–8.

Lynn, Peter. 2003. “PEDAKSI: Methodology for Collecting Data about Survey Non-

Respondents.” Quality & Quantity 37:239–61.

Lynn, Peter, and Paul Clarke. 2002. “Separating Refusal Bias and Non‐Contact Bias:

Evidence from UK National Surveys.” Journal of the Royal Statistical Society: Series D

(The Statistician) 51:319–33.

McCarthy, Douglas B., Debora Shatin, Carol R. Drinkard, John H. Kleinman, and Jacqueline

S. Gardner. 1999. “Medical Records and Privacy: Empirical Effects of Legislation.”

Health Services Research 34(Part II):417–25.

Michaud, Sylvie, David Dolson, Donna Adams, and Martin Renaud. 1995. “Combining

Administrative and Survey Data to Reduce Respondent Burden in Longitudinal Surveys.”

Paper presented at the Joint Statistical Meetings of the American Statistical Association,

Orlando, FL, August. <

http://www.amstat.org/sections/srms/Proceedings/papers/1995_002.pdf>

Mostafa, Tarek. 2015. “Variation within Households in Consent to Link Survey Data to

Administrative Records: Evidence from the UK Millennium Cohort Study,” International

Journal of Social Research Methodology, advance access online.

Murdoch, Maureen, Diane M. Pietila, and Melissa R. Partin. 2010. “Randomized Trial

Showed that an ‘Embedded’ Survey Strategy Optimized Authorization Rates Compared

with Two ‘After Survey’ Strategies in Veterans with PTSD.” Journal of Clinical

Epidemiology 63:665–71.

Olson, Janice A. 1999. “Linkages with Data from Social Security Administrative Records in

the Health and Retirement Study.” Social Security Bulletin 62:73–85.

Olson, Kristen. 2006. “Survey Participation, Nonresponse Bias, Measurement Error Bias, and

Total Bias.” Public Opinion Quarterly 70:737–58.

Olson, Kristen. 2013a. “Paradata for Nonresponse Adjustment.” The Annals of the American

Academy of Political and Social Science 645:142–70.

Olson, Kristen. 2013b. “Do Non‐Response Follow‐Ups Improve or Reduce Data Quality?: A

Review of the Existing Literature.” Journal of the Royal Statistical Society: Series A

(Statistics in Society) 176:129–45.

Olson, Kristen, James M. Lepkowski, and David H. Garabrant. 2011. “An Experimental

Examination of the Content of Persuasion Letters on Nonresponse Rates and Survey

Estimates in a Nonresponse Follow-Up Study.” Survey Research Methods 5:21–26.

Peytchev, Andy, Rodney K. Baxter, and Lisa R. Carley-Baxter. 2009. “Not All Survey Effort

is Equal Reduction of Nonresponse Bias and Nonresponse Error.” Public Opinion

Quarterly 73:785-806.

Peytchev, Andy, and Kristen Olson. 2007. “Using Interviewer Observations to Improve

Nonresponse Adjustments: NES 2004.” Paper presented at the Joint Statistical Meetings

of the American Statistical Association, Salt Lake City, UT, August. <

https://www.amstat.org/sections/srms/proceedings/y2007/Files/JSM2007-000695.pdf>

Sakshaug, Joseph W., Mick P. Couper, Mary Beth Ofstedal, and David. R. Weir. 2012.

“Linking Survey and Administrative Records Mechanisms of Consent.” Sociological

Methods & Research 41:535–69.

Sakshaug, Joseph W., and Martina Huber. 2016. “An Evaluation of Panel Nonresponse and

Linkage Consent Bias in a Survey of Employees in Germany.” Journal of Survey

Statistics and Methodology 4:71-93.

Sakshaug, Joseph W., and Frauke Kreuter. 2011. “Using Paradata and Other Auxiliary Data

to Examine Mode Switch Nonresponse in a ‘Recruit-and-Switch’ Telephone Survey.”

Journal of Official Statistics 27:339–57.

Sakshaug, Joseph W., and Frauke Kreuter. 2012. “Assessing the Magnitude of Non-Consent

Biases in Linked Survey and Administrative Data.” Survey Research Methods 6:113–22.

Sakshaug, Joseph, Valerie Tutz, and Frauke Kreuter. 2013. “Placement, Wording, and

Interviewers: Identifying Correlates of Consent to Link Survey and Administrative data.”

Survey Research Methods 7:133–144.

Sala, Emanuela, Jonathan Burton, and Gundi Knies. 2012. “Correlates of Obtaining Informed

Consent to Data Linkage: Respondent, Interview, and Interviewer Characteristics.”

Sociological Methods & Research 41:414–39.

Sala, Emanuela, Gundi Knies, and Jonathon Burton. 2014. “Propensity to Consent to Data

Linkage: Experimental Evidence on the Role of Three Survey Design Features in a UK

Longitudinal Panel.” International Journal of Social Research Methodology 17:455–73.

Schouten, Barry. 2007. “A Follow-Up of Nonresponse in the Dutch Labour Force Survey.”

Discussion Paper 07004, Statistics Netherlands.

<http://www.cbs.nl/NR/rdonlyres/22AF26C1-C4D7-4DBA-9FC0-EFB7A6682B0E/

0/200704x10pub.pdf>

Smith, Tom W. 2011. “The Report of the International Workshop on Using Multi-Level Data

from Sample Frames, Auxiliary Databases, Paradata and Related Sources to Detect and

Adjust for Nonresponse Bias in Surveys.” International Journal of Public Opinion

Research 23:389–402.

Smith, Tom W., and Jibum Kim. 2013. “An Assessment of the Multi-Level Integrated

Database Approach.” The Annals of the American Academy of Political and Social

Science 645:185–221.

Stone, Celeste, Harmoni Noel, and David Weir. 2013. “Obtaining Administrative Record

Linkage Consent by Mail: Examining the Impact of Incentives and Telephone Follow-

Up.” Paper Presented at the 68th Annual Conference of the American Association for

Public Opinion Research, Boston, MA, May.

<https://www.amstat.org/sections/srms/proceedings/y2013/files/400279_500768.pdf>

Stoop, Ineke, Hideko Matsuo, Achim Koch, and Jaak Billiet. 2010. “Paradata in the European

Social Survey: Studying Nonresponse and Adjusting for Bias.” Paper presented at the

Joint Statistical Meetings of the American Statistical Association, Vancouver, BC,

August. <

https://www.amstat.org/sections/srms/proceedings/y2010/Files/306145_55947.pdf>

Walsh, Rachel, James Dahlhamer, and Nancy Bates. 2013. “Assessing Interviewer

Observations in the NHIS.” Paper Presented at the Joint Statistical Meetings of the

American Statistical Association, Montréal, Québec, Canada, August.

West, Brady T., Frauke Kreuter, and Ursula Jaenichen. 2013. “‘Interviewer’ Effects in Face-

to-Face Surveys: A Function of Sampling, Measurement Error, or Nonresponse?” Journal

of Official Statistics 29:277-297.

Voogt, Robert. J. J., and Hetty van Kempen. 2002. “Nonresponse Bias and Stimulus Effects

in the Dutch National Election Study.” Quality & Quantity 36: 325–45.

Ziegenfuss, Jeanette Y., Kelly R. Burmeister, Ann Haris, Stefan D. Holubar, and Timothy J.

Beebe. 2012. “Telephone Follow-Up to a Mail Survey: When to Offer an Interview

Compared to a Reminder Call.” BMC Medical Research Methodology 12:32

Table 1. Row Percentage and Standard Errors (SE; in Parentheses) of Mail Nonresponse Follow-Up (NRFU) Respondents and NRFU Respondents with Consent, by CATI Nonresponse Outcome

Group D Group E Group G

CATI Nonresponse Outcome

NRFU Sample Size(N=3,922)

NRFU Response% (SE)

(N=539)

NRFU Response w/ Consent

% (SE)(N=501)

Χ2-statistic;p-valuea

Type of nonresponse Refusal Noncontact

1,1902,732

12.86(0.97)

14.13(0.67)

11.85(0.94)

13.18(0.65)

Χ2 = 1.31p = 0.252

Num. of call attempts 0 (invalid number) 1-2 3-7 8-14 15+

874952629751716

13.16(1.14)

13.13(1.10)

14.15(1.39)

13.58(1.25)

15.08(1.34)

12.01(1.10)

12.40(1.07)

13.04(1.34)

13.05(1.23)

13.69(1.29)

Χ2 = 1.20p = 0.878b

Overall -- 13.74(0.55) 12.77(0.53) --a The Χ2-statistic tests for significant differences between the percentage of NRFU response with consent (Group G) and the CATI nonresponse outcomes.b The Χ2-test was also performed on an uncategorized version of the number of call attempts variable which yielded a Χ2-statistic of 0.49 and a p-value of 0.484.

Table 2. Column Percentages (or Means, where denoted) and Standard Errors (in Parentheses) of Administrative Characteristics for the CATI Survey and Mail Nonresponse Follow-Up (NRFU) Survey, by Relevant Study Groups

CATI Survey Mail NRFU Survey

Group A Group B Group CTest 1

(B vs. C) Group D Group G Groups F+HTest 2

(F+H vs. G)Test 3

(B vs. G)

Administrative characteristics

Overall Study Sample

% / Mean (SE)(N=12,400)

CATI Respondents

% / Mean (SE)(N=2,400)

CATI Nonrespondents% / Mean (SE)

(N=10,000)Χ2-Testp-value

NRFU Sample% / Mean (SE)

(N=3,922)

NRFU Respondents w/ Consent

% / Mean (SE)(N=501)

NRFU Nonrespondents

+ NRFU Respondents w/o Consent

% / Mean (SE)(N=3,421)

Χ2-Testp-value

Χ2-Testp-value

Age (in years) 28 or younger 29-41 42-54 55+ Continuous (mean)

17.20(0.34)

30.09(0.41)

31.59(0.42)

21.12(0.37)

43.38(0.12)

18.04(0.79)

25.42(0.89)

34.04(0.97)

22.50(0.85)

43.93(0.28)

17.00(0.38)

31.21(0.46)

31.00(0.46)

20.79(0.41)

43.24(0.13)

0.000

0.024

16.85(0.60)

30.62(0.74)

31.90(0.74)

20.63(0.65)

43.42(0.21)

11.78(1.44)

23.35(1.89)

34.13(2.12)

30.74(2.06)

47.33(0.61)

17.60(0.65)

31.69(0.80)

31.57(0.80)

19.15(0.67)

42.85(0.23)

0.000

0.000

0.000

0.000Male 51.30(0.45) 47.38(1.02) 52.24(0.50) 0.000 52.45(0.80) 44.31(2.22) 53.64(0.85) 0.000 0.211Foreign citizen 10.19(0.22) 5.38(0.46) 11.35(0.32) 0.000 10.45(0.49) 6.59(1.11) 11.02(0.54) 0.002 0.283Education Secondary University entrance College/University Missing

52.59(0.45)

5.61(0.21)

7.52(0.24)

34.28(0.43)

53.42(1.02)

5.67(0.47)

8.42(0.57)

32.50(0.96)

52.39(0.50)

5.60(0.23)

7.30(0.26)

34.71(0.48)

0.095 53.39(0.80)

5.64(0.37)

6.96(0.41)

34.01(0.76)

55.09(2.23)

5.99(1.06)

6.39(1.09)

32.54(2.10)

53.14(0.85)

5.58(0.39)

7.05(0.44)

34.23(0.81)

0.782 0.492

Ever part-time employed 46.98(0.45) 50.84(1.03) 46.05(0.50) 0.000 47.44(0.47) 46.77(2.24) 47.53(0.86) 0.752 0.099Ever full-time employed 88.97(0.28) 90.29(0.61) 88.64(0.32) 0.021 89.00(0.50) 89.72(1.37) 88.89(0.89) 0.583 0.695Monthly earnings (in euros) from last full-time joba

0-600 601-1700 1701-2800 2801+ Continuous (mean)

23.78(0.41)

29.36(0.44)

22.27(0.40)

24.47(0.41)

1880.30(14.71)

24.21(0.93)

31.64(1.01)

21.40(0.89)

22.71(0.91)

1806.11(32.50)

23.67(0.46)

28.80(0.49)

22.48(0.45)

24.90(0.46)

1898.56(16.49)

0.027

0.012

23.32(0.72)

28.56(0.77)

22.59(0.71)

25.36(0.74)

1923.01(26.40)

21.45(1.95)

27.09(2.11)

21.22(1.95)

29.80(2.18)

2094.03(77.84)

23.60(0.78)

28.78(0.83)

22.79(0.77)

24.70(0.79)

1897.69(28.01)

0.136

0.014

0.010

0.000Monthly earnings (in euros) from last part-time joba

0-300 301-700 701-1400 1401+ Continuous (mean)

22.80(0.56)

24.00(0.57)

28.75(0.60)

24.35(0.57)

988.78(12.13)

20.96(1.17)

21.54(1.18)

32.31(1.35)

25.02(1.25)

1029.32(25.92)

23.29(0.63)

24.66(0.64)

27.80(0.67)

24.17(0.64)

977.95(13.72)

0.004

0.087

22.19(0.97)

23.77(0.99)

29.55(1.07)

24.32(1.00)

997.23(21.44)

20.69(2.67)

17.24(2.49)

29.74(3.01)

31.47(3.06)

1164.34(68.08)

22.41(1.04)

24.72(1.08)

29.53(1.14)

23.28(1.06)

973.03(22.42)

0.014

0.004

0.154

0.047Unemployment benefit in 2010 8.88(0.26) 12.88(0.68) 7.92(0.27) 0.000 8.16(0.44) 7.59(1.18) 8.24(0.47) 0.615 0.001Income assistance in 2010 30.36(0.41) 33.08(0.96) 29.71(0.46) 0.001 28.94(0.72) 24.75(1.93) 29.55(0.78) 0.027 0.000a The case base for the full-time and part-time monthly income variables is less than 12,400 due to the requirement of ever being employed as such.

Table 3. Estimates of Percentage Relative Nonresponse Bias and Standard Errors (SE; in Parentheses) for Administrative Characteristics Before and After Mail Nonresponse Follow-Up (NRFU) Survey, and Chi-Squared Test Comparing CATI Respondents Plus NRFU Respondents with Consent versus All Remaining CATI Nonrespondents

Before Mail NRFU Survey After Mail NRFU Survey

Administrative characteristics

CATI Respondents (Group B) Compared to Overall Study Sample (Group A)

CATI Respondents (Group B) + NRFU

Respondents w/ Consent (Group G) Compared to Overall Study Sample

(Group A)

CATI Respondents (Group B) + NRFU Respondents

w/ Consent (Group G) vs. CATI Nonrespondents

(Group C) – NRFU Respondents w/ Consent

(Group G)Χ2-Testp-value

Age (in years) 28 or younger 29-41 42-54 55+ Continuous

4.88(4.59)

-15.52(2.96)

7.76(3.07)

6.53(4.02)

1.27(0.65)

-1.40(4.07)

-16.72(2.69)

7.82(2.79)

13.26(3.74)

2.63(0.58)

0.000

0.000Male -7.64(1.99) -8.67(1.81) 0.000Foreign -47.20(4.51) -45.24(4.22) 0.000Education Secondary University entrance College/University Missing

1.58(1.94)

1.07(8.38)

11.97(7.58)

-5.19(2.80)

2.13(1.77)

1.96(7.66)

7.31(6.78)

-5.16(2.54)

0.112

Ever part-time employed 8.22(2.19) 6.73(1.98) 0.000Ever full-time employed 1.48(0.69) 1.38(0.63) 0.016Monthly earnings (in euros) from last full-time job 0-600 601-1700 1701-2800 2801+ Continuous

1.81(3.91)

7.62(3.44)

-3.98(3.98)

-7.31(3.71)

-3.95(1.73)

-0.21(3.53)

4.97(3.10)

-3.98(3.63)

-2.20(3.43)

-1.32(1.60)

0.256

0.345Monthly earnings (in euros) from last part-time job 0-300 301-700 701-1400 1401+

-8.07(5.13)

-10.07(4.95)

12.47(4.69)

2.67(5.13)

-8.25(4.69)

-12.98(4.45)

11.25(4.27)

7.22(4.76)

0.000

Continuous 4.10(2.62) 6.30(2.47) 0.003Unemployment benefit in 2010 45.05(7.66) 34.68(6.76) 0.000Income assistance in 2010 8.96(3.16) 4.22(2.83) 0.086