the e ect of survey mode on nonresponse and measurement error bias…€¦ · · 2013-05-05a...

The E�ect of Survey Mode on Nonresponse and Measurement

Error Bias. A Validation Study

Barbara Felderer∗ & Antje Kirchner∗

Ph.D. Candidates, Survey Methodology, Ludwig-Maximilians-Universität, Munich

April 30, 2013

Unpublished manuscript, please do not cite.

Abstract

In order to obtain unbiased estimates from survey interviews, it is important that the

data is of good quality: i.e. representativity of the survey respondents and variables that

are free of measurement error. Using administrative records and survey data, the main

questions we address concern the di�erential nonresponse bias between the telephone and

the web mode and whether these modes lead to di�erential measurement error bias.

In an experimental setting we randomly assigned respondents to either telephone (n=2,400)

or web mode (n=1,082). Because the sampled persons were selected from German admin-

istrative records, record data are available for all sample units to study the bias due to

nonresponse and measurement error. Hence, we can assess nonresponse bias by comparing

the statistics from both modes against the known population value. Similarly, we compare

survey values and administrative records for the group of respondents to compute measure-

ment error bias directly.

First, based on administrative data for respondents and nonrespondents, our paper com-

pares nonresponse bias for statistics in the telephone mode to those obtained in the web

mode. Empirical analysis con�rm a di�erential sample composition resulting in systemat-

ically di�erent nonresponse bias between the two modes. Second, we assess the amount of

measurement error bias for both modes. We continue with a discussion of whether mode

speci�c di�erences, with respect to nonresponse bias and measurement error bias, compen-

sate or reinforce each other with respect to the total bias.

Keywords: mode e�ects, telephone survey, web survey, nonresponse bias, measurement

error bias, total bias

∗With conceptual input from Frauke Kreuter.

1

The E�ect of Survey Mode on Nonresponse and Measurement Error Bias. A Validation Study

1 Introduction

When designing a survey, researchers are confronted with manifold decisions, given budget

constraints and the goal to keep the total survey error at a minimum. It is meanwhile a com-

mon practice for surveys to use multiple modes to collect data in order to reduce survey costs,

while at the same time increasing participation (de Leeuw, 2005; Biemer and Lyberg, 2003).

Combining the strengths of di�erent survey modes, a multi mode approach should thus result

in higher overall data quality, i.e. a higher response rate, a more representative sample and

higher response accuracy. The main interest of the following paper is to assess data quality

comparing two di�erent - parallel - survey modes using validation data.1

In general, data quality and thus survey estimates, are a�ected by a range of errors that occur

at various stages along the survey process: e.g. coverage error, sampling error, nonresponse

error, measurement error and adjustment error (Groves et al., 2004). Although survey modes

might in�uence all of these error sources in this paper we speci�cally focus on nonresponse and

measurement error.

Several reasons, such as no contact, refusal, or incapacity by individuals can lead to nonre-

sponse to the survey request (Dillman et al., 2002, 6). These mechanisms di�er across survey

modes: People might either be unwilling or not able to participate in a particular survey mode.

Assuming access to the respective technology, less educated people might be willing to partici-

pate in a telephone study while the same people will likely refuse participation in an otherwise

equivalent web survey. Thus, it can be reasonably assumed that di�erent means of establishing

contact and conducting the interview, may lead to di�erences in sample composition and bias

due to di�erential contact and cooperation rates by certain subgroups (de Leeuw et al., 2008,

300). Error due to nonresponse then causes biased survey statistics if the selection mecha-

nism is not at random but related to variables of interest (Biemer, 2010; Kreuter et al., 2010).

Nonresponse bias is thus a function of the nonresponse rate and the systematic di�erence of

respondents and nonrespondents with respect to the variable under study (Groves and Couper,

1998, 3).

Measurement error on the other hand results if the respondents' survey report di�ers from an

(unobserved) true value (for a review see for example Biemer and Lyberg (2003)). Respondents

might either unintentionally provide (the interviewer with) a false answer, while they might also

consciously decide to give an incorrect response. If this process occurs at random, estimates will

be unbiased. However, estimates will be less precise due to the additional random component.

1Acknowledgements: This research is part of a larger research project directed by Frauke Kreuter at theInstitute for Employment Research, Nürnberg. The IAB project team further included Stephanie Eckman,Barbara Felderer, Antje Kirchner, and Joseph Sakshaug.

Felderer, Kirchner May 5, 2013 2


If the misreporting mechanism is related to the variable of interest, this will result in biased

estimates. Processes leading to measurement error bias di�er across survey modes. While we

would expect to see little di�erence for neutral, factual items, that are not prone to misreport-

ing, survey mode should in�uence measurement accuracy for sensitive, socially (un)desirable

items (Kreuter et al., 2008; Sakshaug et al., 2010). While a respondent might choose to give a

correct answer to a sensitive question in the web mode, she might decide di�erently in a com-

parable telephone mode with an interviewer present (Kreuter et al., 2008). In order to evaluate

measurement error bias, ideally the true value is known and then compared to the survey report.

Regarding the relationship of measurement error and nonresponse bias, several theoretical

models specify ways in which these two might relate. The `common cause model' suggested by

Olson (2007) proposes that variables in�uencing nonresponse also in�uence response accuracy.

An example for one such cause mentioned in the study is the �latent trait �motivation� (Bollinger

and David 2001; Cannell and Fowler 1963). People who possess high values on this latent trait

are likely to participate in the surveys and are also likely to do the hard work of being a survey

respondent� (Olson, 2007, 133). Another model suggested by Olson (2007) is the measurement

process model in which protocol decisions in�uence measurement error. One example provided

by Olson for this model is an interviewer dealing with a reluctant respondent. This perceived

reluctance might lead the interviewer to probing di�erently and incompletely, rushing through

questions, thus inducing measurement error and bias. For an extensive overview regarding

the potential relationships between both error sources see Olson (2007). The purpose of this

paper is not an analysis of the common causes or the measurement process model, but �rst

and foremost an assessment of the relative contribution of each error source to the total bias.

We expect questions on socio-demographic information to be particularly biased with respect

to nonresponse (O'Neill and Dixon, 2005; Abraham et al., 2006; Groves, 2006; Letourneau and

Zbikowski, 2008) esp. in the web survey (Fricker et al., 2005). Other recent studies comparing

web and telephone mode that use validation data suggest that sensitive items are reported

more accurately in the web mode. Measurement error dominated nonresponse error for these

variables, while nonresponse error tended to be larger for neutral and socially desirable items

compared to measurement error bias (Kreuter et al., 2008; Sakshaug et al., 2010). For the most

part Sakshaug et al. (2010) �nd, however, that the di�erent error sources reinforce each other

and do not cancel each other out.

To be able to analyze the e�ect of design choices � such as the use of a particular data collec-

tion mode on certain survey bias components � other error sources have to be kept constant. In

order to satisfy this requirement, we conducted an experimental study assigning sampled cases

to either of two data collection modes � web or telephone � and keep the survey as similar as

possible.



Additionally, to assess the relative contribution of each error source and to avoid confounding

of error sources, auxiliary information is necessary. Ideally this information is available on the

sampling frame for each sample unit. Because the sampled persons in our study were randomly

selected from German administrative records, auxiliary information is available for all sample

units. Furthermore, using record data, the true values are known for the respondents and can

be compared to the respective survey reports. Thus, with our speci�c design it is possible to

disentangle bias due to nonresponse and measurement error for each survey mode. The avail-

ability of such administrative data as a gold standard is quite rare in the literature on mode

e�ects comparing web and telephone (McCabe et al., 2002; Sax et al., 2003; Kreuter et al., 2008;

Dillman et al., 2009; Sakshaug et al., 2010), and allows us a unique opportunity to evaluate

data accuracy across modes.2

Further, prior studies analyzing web survey mode e�ects either consider the total bias of a

survey statistic or focus on single sources of error (McCabe2002, Sax2003, Sanders2007, Dill-

man2009, Stephenson2011). The two validation studies by Kreuter et al. (2008) and Sakshaug

et al. (2010) are an exception. Both studies especially focus on the interaction of di�erent

error sources across survey modes for di�erent survey items. Further, existing studies typically

analyze nonresponse or measurement error bias for mean statistics of certain survey items but

do not assess bias in distributions. The most serious limitation of existing studies, however, is

their limited generalizability to rather speci�c populations (such as student alumnis (Kreuter

et al., 2008; Sakshaug et al., 2010) or high income individuals with one child (Dillman et al.,

2009)). The contribution of our study is �rst to study the interaction of two sources of bias on

survey statistics using neutral, socially undesirable and sensitive items. Second, we include an

assessment beyond the analysis of the bias of mean statistics and also compare distributions

for two metric items. Most importantly, we use a strati�ed random sample of the general adult

population in Germany. Thus more generalizable inferences can be drawn regarding the mode

choice.

In summary, we address the following two questions:

First, what is the (relative) contribution of nonresponse and measurement error bias to

the overall bias in the survey estimates across modes?

Second, does the relationship between those error sources di�er for types of variables and

modes?

2Some studies investigating mode e�ects use panel surveys as a sampling frame, thus limiting generalizability(Sax et al., 2003; Braunsberger et al., 2007; Chang and Krosnick, 2009).



The remainder of the article is organized as follows: Section 2 describes our empirical study,

the administrative records and potential limitations with respect to the data. In section 3 we

describe our methodological approach to analyze the data while section 4 presents the results

of our study. Section 5 discusses the results and limitations.

2 Data and Methods

We carried out an experiment randomly assigning respondents to either phone or web survey

mode to assess bias due to nonresponse and measurement error. We use survey data from a

population-based telephone and web survey, that was commissioned by Institute for Employ-

ment Research (IAB), Nuremberg, Germany and carried out by the LINK institute.

2.1 Survey Data

Sampling and Experimental Design: A strati�ed random sample of the general adult popula-

tion in Germany was drawn from the Integrated Employment Biographies (IEB) maintained

by the German Federal Employment Agency (FEA). This register combines information from

various sources on employees3 and basic income support recipients ("Unemployment Bene�t

II", short UB II) in Germany. This sampling frame is comprehensive, up to date (with only a

short time lag) and accurate, since it contains payment relevant information: Employment data

stem from employer reports to social security agencies and are used by the German statutory

pension insurance to calculate pension claims. Similarly the information about bene�t receipt

is used to administer bene�t claims and payments. In addition, addresses and in part telephone

numbers for all sample cases were available on the frame.

Overall a gross sample of 24,236 eligible persons (aged 18 and older) was drawn in June 2011

from the FEA registers according to the following three strata: unemployed, welfare recipients

(UB II) or employed in the reference period 2001 to 2010.4 Within the three strata, respon-

dents were randomly allocated to the two modes: 12,400 cases were randomly assigned to the

telephone survey while 11,836 cases were assigned to the web survey. However, because of the

low quality of telephone numbers a larger share of the employed strata was allocated to the

telephone mode. Table 1 provides an overview of the sample sizes and response rates.

Data Collection Telephone Survey (CATI): The FEA registers contain telephone numbers

(landline and/or mobile numbers) only for about forty percent of the employees and ninety

percent of welfare recipients. Furthermore, it is known from past surveys that some addresses

3More precisely, the register includes all employees who are subject to social security contributions. Self-employed and civil servants are not covered because they do not pay social security contributions.

4A strati�cation plan was developed that divided the population into these three mutually exclusive strata.Further information regarding strati�cation is available from the authors upon request.



Telephone Survey Web Survey

gross sample N 12,400 11,836net sample (contactability) 9,332 10,525completed interviews n 2,400 1,082

Response Rate (base gross) 19.35 % 9.01%

Table 1: Response Rates Across Modes of Data Collection

and some of the telephone numbers on the frame are out of date. For the telephone component

of the survey we therefore tried to complete the numbers using public telephone directories and

internet research.

For 10,455 out of 12,400 cases a telephone number was available. During �eldwork about 11

percent of the phone numbers turned out to be invalid, and could not be replaced by a working

phone number from the public directories, or ineligible. Among those persons who could be

contacted (n=9,332), the target of 2,400 completed interviews could be met.5. During �eldwork

at least 20 contact attempts were made per case at varying points in time. However, we did

not attempt speci�c refusal conversion once a case expressed some mild form of refusal. Due to

the large proportion of missing or invalid phone numbers, the overall response rate was 19.35

percent (RR1 according to AAPOR 2011).

Prior to �eldwork all sampled cases received an advance letter inviting them to participate in

the government survey "Work and Consumption in Germany". Information on the voluntary

nature of the survey, the survey topic, survey sponsorship, a toll free contact number and e-mail

contact were included. Fieldwork was conducted in the months of August to October 2011.

The questionnaire contained questions relating to employment biographies, personality traits

of the respondents and social deprivation. Some experiments were conducted regarding the

exact question format, however, these are of no e�ect to the mode comparison Kreuter et al.

(2012). The average survey completion time was 21 minutes.

Data Collection Web Survey (Web): We mailed an invitation letter to all sampled cases of

the web survey. This letter was kept as similar as possible to the advance letter used for the

telephone survey ("Work and Consumption in Germany"). The only di�erence was the men-

tioning of the link to the web survey and a unique user name and password for login. Overall

1,311 letters were returned to sender due to an incorrect address. Thus those respondents never

received the invitation to participate. In order to reduce undercoverage and improve response

rates the advance letter also o�ered the opportunity to call a toll free number to conduct a

5Overall we had 194 partial interviews, i.e. those respondents who did not complete the main substantiveexperiments of our multi purpose survey. For substantive experiments see for example (Kreuter et al., 2012)



shortened version of the interview on the telephone and a conditional incentive of three Euro.

Out of 10,525 cases 312 people made use of the hotline. Out of those calls 161 called to sched-

ule an appointment for a telephone interview resulting in an additional 132 interviews. The

shortened telephone interviews are not part of our analyses.

Respondents received a set of questions related to their employment biographies which were

equivalent to those in the telephone mode (with only slight modi�cations due to the change in

mode). However, the web survey did not contain a section on social deprivation. Additionally

experiments that were conducted in the telephone survey were replicated in the web survey

as far as possible. The web survey was shorter than the telephone survey with only about 15

minutes average completion time.

Fieldwork was conducted from mid February to mid April 2012. Two weeks after the start

of �eldwork a reminder was sent to all cases. Overall 1,082 people completed the web survey.

This includes 14 partial interviews: These respondents completed all substantive parts up to

the socio-demographic section. Those 206 cases who only partially completed the substantive

parts of the online questionnaire were excluded from the analyses. The overall response rate in

the web survey was 9.01 percent.

2.2 Administrative Data

We use administrative data from the IEB �le provided by the Research Data Center of the

FEA.6 This register data contains detailed employment (e.g type of employment, income) and

welfare bene�t recipient information (e.g type of bene�t) for all sample units, i.e. respondents

and nonrespondents to the survey. Data provided in the IEB register is spell based so that

entire employment histories are available for each individual, starting from 1975, as well as

complementary information on times of unemployment and welfare receipt.

We focus exclusively on those items that are measured identically in both survey modes, and

that are available as well as conceptually equivalent in the administrative data. With respect

to the administrative data, we only use variables to assess nonresponse bias and measurement

error that are known to be accurate and complete and can thus serve as gold standard. As

a general rule, all data relevant to payments (pensions, welfare etc.), i.e. the primary use of

the system, are known to be of very good data quality. IEB data is found to be very reliable

concerning socio-demographic characteristics, such as gender or age, as well as employment

related characteristics, such as employment status and type of employment, wages, as well as

bene�t receipt (Jacobebbinghaus and Seth, 2007).

6IAB Integrierte Erwerbsbiogra�en (IEB) V10.00.00, Nürnberg 2012



Thus our analysis focuses on the following variables: gender (0 male, 1 female), mean age (as

well as categories: 18 - 29, 30 - 39, 40 - 49, 50 - 59, >=60), school education (unknown degree,

lower and intermediate secondary degree, higher secondary degree, teriary degree)7, nationality

(0 German, 1 Non-German)8, currently employed (0 no, 1 yes), type of employment (marginal

employment (income <=400 Euro), regular employment (income >400 Euro)), current receipt

of basic income support (UB II) (0 no, 1 yes), past receipt of UB II (past 12 months) and mean

monthly income if currently employed (in Euro and terciles).9

Data used for the analyses of socio-demographic variables was always extracted from either

the last valid employment spell or, in case of an ongoing spell, in the respective interview month

for respondents. The date of the last interview in either mode is taken as the reference date

for nonrespondents. Due to data availability of administrative data, income in the web survey

is available only up to 31.12.2011 and will be compared to survey data from 02.2012. 10

2.3 Analytical Sample and Experimental Design

For our analyses of nonresponse bias, we focus on those individuals only for whom we have valid

contact information, i.e. those who had a chance to participate in the survey: In the telephone

survey that is all persons with valid telephone numbers, for the web survey those with valid

addresses. All (contacted) target persons were eligible. Nonresponse is further distinguished

into noncontact, i.e. those people we were never able to contact on the phone, and refusal,

i.e. noncooperation given contact. In the web survey, all nonresponse is assumed to be due to

refusal, since 'contact' was established once the invitation letter was delivered.

2.4 Statistical Analysis

The following bias analyses compare survey mean statistics (and distributions) for the items

introduced above, and administrative records in both modes for:

7Education is less reliably measured for employment spells, thus an 'unknown' category has be included forthe analyses. Also, education is not included in the measurement bias analysis

8Conceptually the administrative data captures respondent nationality. This does not match the survey datawhich measures migration background of the parents or grandparents. Thus nationality is only included inthe nonresponse analyses but not in the measurement bias analysis.

9In Germany, respondents think in terms of monthly income and not yearly income. Thus the survey itemsask for monthly income. However, labor income in the administrative records is captured only as the totalgross income in a given employment spell (typically one year). Thus monthly income has to be derived fromthis measure. The basic assumption being that all income is equally distributed over the months of a certainspell. Also, income is top coded in the administrative data, the limit being a yearly income of 69,000 Euro.Since this a�ects all administrative data equally and survey mode was randomly allocated, inferences withrespect to the relative comparison across modes are still valid.

10Data covering the entire web survey �eldwork period will be available as of September 2013.



a) full sample administrative data

b) respondent sample administrative data

c) respondent sample survey data

More precisely, we base our comparison on a) the full sample using administrative data to

obtain the 'true parameters' as a reference for the whole sample, b) the respondents using

administrative data to obtain the 'true parameters' as a reference for the respondents, and c)

the respondents, i.e. survey data separately by mode. Statistics are derived from a) the full

sample administrative data and b) the respondent sample administrative data to investigate

nonresponse bias. Similarly we will compare b) the respondent sample administrative data and

c) respondent sample survey data to assess measurement error bias. For an assessment of the

total bias we compare a) the true value from the full sample administrative data and c) the

respondent sample survey data.

For our data the estimation of nonresponse bias is straightforward as information is available

for all sample cases and other error sources can be neglected. Nonresponse bias (nr) is given

as the di�erence of the true sample value according to administrative records (adm) of the full

sample (fs) and the mean computed using the respondents (resp) only.

bias(yadm,nr) = yadm,resp − yadm,fs (1)

In order to compare nonresponse bias between variables and modes the bias is standardized

using the full sample mean obtaining the relative bias (Olson, 2006):

rel.bias(yadmin,nr) =bias(yadm,nr)

yadm,fs∗ 100 (2)

Nonresponse bias will further be decomposed into nonresponse bias due to noncontact (nc)

and refusal (ref). The absolute and relative biases can be estimated analogously.

Similar to the estimation of nonresponse bias, bias due to measurement error (me) is straight-

forward to calculate as the true values are known from the administrative records. Measurement

error is given as the di�erence the mean statistics in the survey data (svy) and the true value

of a statistic according to administrative records for all respondents. This gives us an estimate

of the absolute magnitude of the measurement error bias:



bias(yme) = ysvy,resp − yadm,resp (3)

There is a negligible number of item nonresponse in the survey data in some of the categor-

ical variables. Since we are estimating the proportion of respondents belonging to a certain

category (yes/no), missing information is implicitly treated as a `no' response in the assessment

of measurement bias. Also, to be able to compare measurement error bias estimates across

variables and modes this bias estimate can be standardize with the mean of the respondents

based on the administrative data.

The total bias (tot) for a survey statistic is simply the sum of bias due to nonresponse and

measurement error. The total bias estimate can be standardized analogously to the other statis-

tics to obtain the relative total bias.11

3 Results

We present the results in two parts. In a �rst step bias due to nonresponse error will be anal-

ysed for both modes. Nonresponse bias, i.e. noncontact and refusal bias, will be assessed based

on administrative data for gender, age, education, current employment (y/n and type), current

and past receipt of welfare (UB II) as well as labor income from the current job.

Second, we will assess bias due to measurement error for both modes. Two of the items

investigated for nonresponse bias are conceptually too di�erent in the administrative data

(education, nationality) or missing in the survey (current receipt of UB II) and cannot be com-

pared. Thus only gender, age, current employment (y/n and type of employment), past receipt

of welfare (UB II) as well as labor income from current job are analyzed. Mean estimates are

easily in�uenced by outliers when investigating metric variables. Thus, we will also catego-

rize age and income and report the amount of respondents belonging to these age and income

groups. In addition, we will test the equality of distributions in the survey responses and the

true distribution using a Kolmogorov-Smirnov test.

Last, we will analyze the relative contribution of nonresponse bias and measurement error

bias to the total bias for those items that have a measurement error bias estimate.

11Standard errors for all relative bias estimates are obtained applying a nonparametric bootstrap, based on5000 repeated samples. Con�dence bands are reported according to the empirical quantiles (0.025, 0.975).



3.1 Nonresponse Bias According to Mechanism of Nonresponse

Figure 1 provides an overview of the relative nonresponse bias estimates for the socio-demographic

variables and their decompositions to noncontact and refusal bias for both survey modes.12.

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

Female

Mean Age (Years)

Edu. unknown

Edu. low./int. sec.

Edu. up. sec.

Edu. tert.

Nationality

-50 0 50 100 150

Socio-Demography: Relative Nonresponse Bias (in %)

CATI Web

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

Employed:

Regular empl.

Marginal empl.

Mean Inc. (Euro)

Currently UB II

Past UB II

-40 -20 0 20 40

Substantive Items: Relative Nonresponse Bias (in %)

CATI Web

Figure 1: Relative nonresponse bias for socio-demographic and substantive variables includingcon�dence intervals (0.025, 0.975)

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

ncrefnr

Mean Age (Years)

<=30

30-39

40-49

50-59

>=60

-40 -20 0 20 40

Age: Relative Nonresponse Bias (in %)

CATI Web

nc

ref

nr

nc

ref

nr

nc

ref

nr

nc

ref

nr

Mean Inc. (Euro)

Low Inc.

Middle Inc.

High Inc.

-40 -20 0 20 40

Income: Relative Nonresponse Bias (in %)

CATI Web

Figure 2: Relative nonresponse bias for age and income distributions including con�dence in-tervals (0.025, 0.975)

While there is no relative nonresponse bias for gender in the web mode, women are signi�-

cantly overrepresented in the CATI mode. In particular, women are more likely to be contacted

and more likely to participate once they have been contacted.

For the web mode we �nd a signi�cant underestimation of the mean age. More precisely,

�gure 2 shows that the younger age groups are heavily overrepresented (thus only signi�cant

12Tables for relative and absolute bias estimates can be found in appendix A



for the group age 30 to 39) and the older age groups are heavily underrepresented (signi�cantly

for the oldest group of 60 years and older). This pattern is reversed for the CATI mode with

younger cases being under- and older cases being overrepresented in the survey. Overall, age

is less biased in the CATI compared to the web mode. While the relative refusal bias by age

is higher (though not signi�cantly) for the web than the CATI mode, it points in the same

direction for both modes. Part of the large relative nonresponse bias in the CATI mode is also

due to the low contactability of the younger cases (<=39).

For both modes we �nd that the cases with the highest education (university degree) are

heavily overrepresented. The amount of overrepresentation is four times high as high in the

web compared to the CATI mode. Also the share of respondents with a higher secondary

education is overestimated in the web mode, whereas lower educated cases tend to be under-

estimated. There is no considerable refusal bias for education in both modes. Only the group

with the highest education is signi�cantly more easily contacted than the other groups within

the CATI mode. Overall, relative nonresponse bias concerning education is much larger in the

web than in the CATI mode.

Employment status tends to be biased in the same direction for web and CATI (see �gure

1) but bias again is much larger for the web mode. Employed people and especially cases hav-

ing a regular job are signi�cantly overrepresented in both surveys. Also, for the CATI mode,

marginally employed cases are more likely to be contacted compared to regularly employed in-

dividuals. In both modes, employed cases, especially with regular employment are more willing

to participate in the survey once they have been contacted.

The mean income from employment is overestimated in the web survey. In this mode, the

proportion of cases belonging to the lowest income category is under- and the proportion of

cases belonging to the highest income category is overestimated (see �gure 2). Although we

�nd a signi�cant relative noncontact bias for the middle and high income groups, there is no

signi�cant relative nonresponse bias in the CATI survey.

Both, the proportion of current and past welfare recipients is highly underestimated in the

web mode (�gure 1). Again, there is signi�cant relative noncontact bias in the CATI survey:

Current and past recipients being less likely to be contacted. Here, the relative noncontact

bias for current bene�t receipt is compensated by an opposing relative refusal bias. Thus the

resulting relative nonresponse bias is not signi�cant for this item. Even though the relative

refusal bias can compensate some of the relative noncontact bias for the past receipt of bene�ts,

we still observe a signi�cant underestimation of former bene�t receipt in the CATI mode. The

relative nonresponse bias for past unemployment bene�t receipt is more than six times as high



in the web than in the phone mode.

Overall, the relative nonresponse bias estimates for the web survey are larger in magnitude

compared to the telephone mode. Also, due to the smaller sample size, con�dence intervals are

much broader in the web mode.

Taking a closer look at the individual sources of nonresponse bias (i.e. bias due to noncon-

tact and refusal) that are statistically signi�cant, we can see that for the CATI mode, much

of this overall nonresponse bias is driven by bias due to noncontact. While noncontact and

nonresponse bias enforce each other and lead to a signi�cant overall nonresponse bias for gen-

der, age (30-40, 50-60), education (tertiary degree) and currently employed (especially regular

employed), they oppose each other � thus reducing overall nonresponse bias � for mean, middle

and high income as well as past and current receipt of UB II.

To summarize our results, we do see that the telephone sample is less selective and nonre-

sponse bias estimates are generally lower compared to those in the web survey.

3.2 Measurement Error Bias and Total Bias

Second, relative bias from measurement error will be assessed for some socio-demographic and

substantive variables, i.e. type of employment, past receipt of UB II and mean labor income

from current job. 13

nr

me

tot

nr

me

tot

Female

Mean Age (Years)

-60 -40 -20 0 20

Socio-Demography: Relative Total Bias (in %)

CATI Web

nr

me

tot

nr

me

tot

nr

me

tot

nr

me

tot

nr

me

tot

Employed:

Regular empl.

Marginal empl.

Mean Inc. (Euro)

Past UB II

-50 0 50

Substantive Items: Relative Total Bias (in %)

CATI Web

Figure 3: Relative total bias for socio-demographic and substantive variables including con�-dence intervals (0.025, 0.975)

Our results for the 'neutral' socio-demographic variables (gender and age) are in line with

13Results including absolute bias estimates of measurement error can be found in appendix B.



nrmetot

nrmetot

nrmetot

nrmetot

nrmetot

nrmetot

Mean Age (Years)

<=30

30-39

40-49

50-59

>=60

-60 -40 -20 0 20 40

Age: Relative Total Bias (in %)

CATI Web

nr

me

tot

nr

me

tot

nr

me

tot

nr

me

tot

Mean Inc. (Euro)

Low Inc.

Middle Inc.

High Inc.

-40 -20 0 20 40

Income: Relative Total Bias (in %)

CATI Web

Figure 4: Relative total bias for age and income distributions including con�dence intervals(0.025, 0.975)

our expectations (see �gures 3 and 4): there is no signi�cant relative measurement error bias

for both modes. Thus, for these variables the relative total bias is essentially the same as the

relative nonresponse bias. There is, however, a nonsigni�cant di�erence between the survey and

administrative mean age in the web survey. This is driven by one far outlier who reported to be

-7987 years old. Only looking at the mean age would be entirely misleading in the web survey.

Comparing the full densities does not show any signi�cant di�erence between administrative

and survey data for any mode (see Appendix C for more details).

Regarding socially (un)desirable characteristics, we do see signi�cant deviations from the true

means. For the CATI mode we observe signi�cant overreporting of regular employment, while

marginal employment is signi�cantly underreported in both modes. The e�ects are smaller in

the web survey compared to the telephone survey (nonsigni�cant) suggesting a higher report-

ing accuracy in the web mode. We attribute these results to social desirability: Telling an

interviewer on the telephone that one has a regular job is more desirable and less of a norm

violation than admitting to be "only" marginally employed in Germany.

Both modes show considerable relative measurement error bias regarding income (�gure 3):

In both modes too few respondents report that they belong to the highest income category

while too many respondents claim to be in the low income group (though only signi�cant for

the CATI mode). The mean income estimation shows only signi�cant measurement error bias

for the web survey. Overall, for both modes the income distribution of respondents' survey

reports and respondents' administrative records di�er signi�cantly (p < 0.05, see Appendix C).

The results for the items regarding labor income, however, need to be treated with care since

the administrative data for the web survey are not yet up to date and need to be reanalyzed

when data is available.



Measurement error bias for past receipt of UB II di�ers in magnitude in both modes (see

�gure 3): While we see signi�cant underreporting in the telephone mode, there is signi�cant

overreporting in the web mode. These results suggest that social desirability concerns from re-

spondents can be alleviated in the web mode. However, the overreporting of past UB II receipt

in the web mode is somewhat puzzling. One explanation for the overreporting of unemploy-

ment bene�ts could be slight di�erences in the questions between the modes. The telephone

survey asked for 'welfare receipt in 2010', i.e. the previous year. Due to the timing of the web

survey (beginning of 2012), the item in the web survey asked for 'welfare receipt in the past

12 months'. The administrative data can exactly di�erentiate these di�ering periods, while we

suspect that some respondents' reports in the web survey su�er from so-called telescoping, a

special form of recall error. This would result in respondents reporting receipt prior to the

12 months reference period - since February 2011 - and thus explain the signi�cant amount of

overreporting (i.e. reports include receipt in January 2011). This potential error from telescop-

ing slightly confounds our results with respect to mode di�erences. Nonetheless, web seems to

outperform telephone for this item since it is able to alleviate social desirability concerns.

To summarize, for socially (un)desirable items the relative measurement error bias tends to

the same direction for both modes and is in line with our expectations. For neutral items

measurement error in both modes is virtually nonexistent. The results in the web mode for

type of employment and labor income, however, need to be treated with care. They are only

preliminary since the administrative data for the web survey are not yet up to date but from

2 months prior to the survey interview. Yet, already with these 'imputed' values measurement

error bias tends to be smaller for employment type and labor income. Thus we are con�dent

that a reanalysis of these indicators will yield even more favorable results for the web survey.

A comparison of the total bias across both modes shows that the relative total bias is sig-

ni�cantly larger for the web mode for the items age (mean age, age 30-39, age >60), currently

employed and labor income (mean income and lowest and highest income group). The remain-

ing items do not di�er signi�cantly across survey modes. These results suggest that the web

survey data are overall more biased compared to the telephone mode.

Taking a closer look at the individual contributions of nonresponse and measurement error

bias to these signi�cant di�erences, we see that nonresponse bias almost always exceeds mea-

surement error bias in magnitude (see appendix B for absolute contributions). One exception

being past receipt of unemployment bene�t in the CATI mode: Relative measurement error

bias is larger than relative nonresponse bias. With respect to directionality of the e�ects �gures

3 and 4 show that nonresponse bias and measurement error bias tend to reinforce each other



(except for past bene�t receipt and the income categories in the web survey).

For the remaining items the total bias does not di�er signi�cantly across modes. However,

also for these items we �nd support for the relationship that nonresponse bias tends to exceed

measurement error bias in magnitude in both modes. Signi�cantly so for gender (CATI), age

(30-40, 50-60 and >60, for both modes), 'regular employment' (web), as well as low and high

income (web).

To summarize our results with respect to total bias, we �nd that the web survey data is

overall more biased compared to the telephone survey. These results are mainly driven by the

nonresponse bias in both modes (reinforced by measurement error bias). For past unemploy-

ment bene�t receipt we �nd that even though there is signi�cantly higher relative nonresponse

and measurement error bias for the web mode, these biases reduce each other. Thus, the relative

total biases do not signi�cantly di�er between the modes.

4 Discussion

Our experiments show that overall web survey mean estimates are more biased compared to the

telephone mode. This result is driven by a larger nonresponse bias enforced by a comparatively

smaller measurement bias. Measurement error bias for sensitive and socially desirable items is

larger in the telephone mode compared to the web mode.

This raises the question how pooling data collection modes or adjusting for nonresponse

would a�ect the overall bias. Further analyses along these lines will be conducted when the

�nal data is available. This is one serious limitation of the study so far. Analyses of measure-

ment error for income and type of employment rely on the assumption that those data collected

in the survey are equivalent to the administrative data of 2 months prior to data collection.

However, the measurement error is already lower for the web survey with this assumption,

thus we are con�dent that the overall conclusions will remain and that results with respect to

measurement error bias in the web survey will be even more favorable.



References

Abraham, K. G., A. Maitland, and S. M. Bianchi (2006). Nonresponse in the american time

use survey. who is missing from the data and how much does it matter? Public Opinion

Quarterly 70(5), 676�703.

Biemer, P. P. (2010). Handbook of Survey Research, Chapter 2 Overview of Design Issues: Total

Survey Error, pp. 27�57. Emerald.

Biemer, P. P. and L. E. Lyberg (2003). Introduction to survey quality. New York: Wiley.

Braunsberger, K., H. Wybenga, and R. Gates (2007). A comparison of reliability between

telephone and web-based surveys. Journal of Business Research 60, 758�764.

Chang, L. and J. A. Krosnick (2009). National surveys via rdd telephone interviewing versus

the internet. comparing sample representativeness and response quality. Public Opinion

Quarterly 73(4), 641�678.

de Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of

O�cial Statistics 21(2), 233�255.

de Leeuw, E. D., J. J. Hox, and D. D. A. (2008). International Handbook of Survey Method-

ology, Chapter 16 Mixed-mode Surveys: When and Why, pp. 299�316. New York/London:

Erlbaum/Taylor & Francis.

Dillman, D. A., J. L. Eltinge, R. M. Groves, and R. J. A. Little (2002). Survey Nonresponse,

Chapter 1 Survey Nonresponse in Design, Data Collection and Analysis, pp. 3�26. New York:

Wiley.

Dillman, D. A., G. Phleps, R. Tortora, K. Swift, J. Kohrell, J. Berck, and B. L. Messer (2009).

Response rate and measurement di�erences in mixed-mode surveys using mail, telephone,

interactive voice response (ivr) and the internet. Social Science Research 38, 1�18.

Fricker, S., M. Galesic, R. Tourangeau, and Y. Ting (2005). An experimental comparison of

web and telephone surveys. Public Opinion Quarterly 6(3), 370�392.

Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public

Opinion Quarterly 70(5), 646�675.

Groves, R. M. and M. Couper (1998). Nonresponse in Household Interview Surveys. Wiley

Series in Probability and Statistics: Survey Methodology Section. New York: Wiley.

Groves, R. M., F. J. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer, and R. Tourangeau

(2004). Survey Methodology. Wiley.



Jacobebbinghaus, P. and S. Seth (2007). Ther german integrated employment biographies

sample iebs. Schmollers Jahrbuch 127, 335�342.

Kreuter, F., S. Eckman, A. Jäckle, A. Kirchner, S. Presser, and R. Tourangeau (2012). Mech-

anisms of misreporting to �tler questions. In American Association of Public Opinion Re-

search, 67th Conference.

Kreuter, F., K. Olson, J. Wagner, T. Yan, T. Ezatti-Rice, C. Casas-Cordero, A. Petychev,

R. M. Groves, and T. Raghuatan (2010). Using proxy measures and other correlates of

survey outcomes to adjust for non-response: examples from multiple surveys. Journal of the

Royal Statistical Society A 173(2), 389�407.

Kreuter, F., S. Presser, and R. Tourangeau (2008). Social desirability bias in cati, ivr, and web

surveys. the e�ects of mode and question sensitivity. Public Opinion Quarterly 72, 847�865.

Letourneau, P. M. and A. A. Zbikowski (2008). Nonresponse in the american time use survey.

In ASA Section on Survey Research Methods, JSM, pp. 1283�1290.

McCabe, S. E., C. J. Boyd, M. Couper, and S. Crawford (2002). Mode e�ects for collecting

alcohol and other drug use data: Web and u.s. mail. Journal of Studies on Alcohol 63(3),

755�761.

Olson, K. (2006). Survey participatoin, nonresponse bias, measurement error bias, and total

bias. Public Opinion Quarterly 70(5), 737�758.

Olson, K. M. (2007). An Investigation of the Nonresponse-Measurement Error Nexus. Ph. D.

thesis, The University of Michigan.

O'Neill, G. and J. Dixon (2005). Nonresponse bias in the american time use survey. In ASA

Section on Survey Research Methods, JSM, pp. 2958�2966.

Sakshaug, J. W., T. Yan, and R. Tourangeau (2010). Nonresponse error, measurement error,

and mode of data collection. Public Opinion Quarterly 74(5), 907�933.

Sax, L. J., S. K. Gilmartin, and A. N. Bryant (2003). Assessing response rates and nonresponse

bias in web and paper surveys. Research in Higher Education 44(4), 409�432.



Appendix: A - Nonresponse Bias



Rel.noncontactbias

Rel.refusalbias

Rel.nonresponse

bias

Variable

CATI

Web

CATI

Web

CATI

Web

Female

1.86

-7.07

3.9

9.06

3.9

(0.6;3.16)

(3.78;10.24)

(-1.92;9.5)

(5.56;12.7)

(-1.68;9.44)

Meanage(years)

3.95

--2.15

-4.97

1.72

-4.97

(3.57;4.33)

(-3.14;-1.22)

(-6.56;-3.39)

(0.63;2.82)

(-6.59;-3.4)

Age≤

30

-14.18

-11.4

7.08

-4.39

7.08

(-17.04;-11.37)

(3.99;18.76)

(-6.43;20.35)

(-11.62;2.6)

(-5.69;20.06)

Age30-39

-11.85

--1.81

23.5

-13.45

23.5

(-14.3

;-9.41)

(-8.47;4.55)

(12.14;34.8)

(-19.61;-7.28)

(12.42;34.75)

Age40-49

0.95

-3.15

0.09

4.13

0.09

(-1.36;3.21)

(-2.52;9.02)

(-9.27;9.76)

(-1.91;10.29)

(-9.48;9.6)

Age50-59

9.29

-2.3

-6.87

11.8

-6.87

(6.86;11.65)

(-3.93;8.33)

(-17.65;4.65)

(4.98;18.87)

(-17.87;3.83)

Age≥

60

21.88

--15.13

-33.07

3.45

-33.07

(19.4

;24.36)

(-21.73;-8.43)

(-44.31;-21.72)

(-4.85;11.73)

(-44.17;-21.63)

Educationunknown

-3.13

-1.62

-9.68

-1.55

-9.68

(-6.35;0)

(-6.5

;9.76)

(-23.18;3.97)

(-9.84;6.98)

(-22.59;3.96)

Educationlower/interm

ediate

secondary

1.25

-2.19

-2.81

3.46

-2.81

(-0.31;2.86)

(-1.98;6.2)

(-10.13;4.48)

(-0.77;7.91)

(-10.16;4.6)

Educationhighersecondary

-1.67

-8.55

38.57

6.74

38.57

(-7.1

;3.56)

(-4.83;22.43)

(13.1

;65.81)

(-7.33;21.37)

(13.21;64)

Educationtertiary

10.29

-11.18

87.54

22.63

87.54

(5.94;14.43)

(-0.5

;22.45)

(61.94;113.23)

(9.18;36.02)

(62.24;113.77)

Nationality

(Germ

an)

1.07

-2.77

5.42

3.87

5.42

(0.66;1.5)

(1.9

;3.63)

(3.98;6.77)

(2.93;4.79)

(4;6.75)

Employed:

1.16

-4.61

13.77

5.82

13.77

(0.09;2.24)

(2.01;7.35)

(9.17;18.23)

(2.93;8.7)

(9.32;18.26)

Regularemployment

0.79

-5.88

19.79

6.71

19.79

(-0.59;2.19)

(2.36;9.37)

(13.52;25.74)

(2.8

;10.58)

(13.75;25.76)

Marginalemployment

6.62

--2.23

-10.39

4.24

-10.39

(2.98;10.16)

-(-11.46;7.07)

(-26.32;5.86)

(-6.01;14.95)

(-26.58;6.34)

Meanincome(Eur)

1.6

--0.82

20.01

0.77

20.01

(0.41;2.8)

(-3.98;2.04)

(14.44;25.61)

(-2.56;4.05)

(14.4

;25.47)

Lowincome

0.24

-1.47

-23.18

1.72

-23.18

(-1.89;2.45)

(-4.18;7.01)

(-31.98;-14.1)

(-4.26;7.67)

(-31.64;-14.16)

Middleincome

-3.58

-2.93

-6.87

-0.75

-6.87

(-5.83;-1.25)

(-2.97;8.7)

(-16.36;2.83)

(-6.87;5.31)

(-16.01;2.6)

Highincome

3.6

--4.55

32.09

-1.12

32.09

(1.16;5.9)

(-10.34;1.05)

(21.27;42.51)

(-7.5

;5.31)

(21.65;43.07)

CurrentreceiptofUBII

-10.2

-11.93

-29.38

0.51

-29.38

(-12.68;-7.79)

(5.27;18.34)

(-39.83;-19.01)

(-5.89;6.91)

(-40.02;-18.7)

Past

receiptofUBII

-11.92

-6.82

-37.88

-5.91

-37.88

(-13.71;-10.06)

(1.81;11.81)

(-45.82;-29.74)

(-10.6

;-1.33)

(-45.68;-29.92)

Table2:

Relativenoncontact

bias,relative

refusalbiasandrelative

nonresponse

biasincludingcon�dence

intervals(0.025,0.975)



Noncontact bias Refusal bias Nonresponse biasVariable CATI Web CATI Web CATI Web

Female 0.009 - 0.0347 0.0194 0.0437 0.0194

Mean age (years) 1.7246 - -0.9754 -2.1944 0.7492 -2.1944Age ≤ 30 -0.0268 - 0.0185 0.0118 -0.0083 0.0118Age 30-39 -0.0268 - -0.0036 0.0526 -0.0304 0.0526Age 40-49 0.0022 - 0.0075 0.0002 0.0097 0.0002Age 50-59 0.0188 - 0.0051 -0.0139 0.0238 -0.0139Age ≥ 60 0.0325 - -0.0274 -0.0507 0.0051 -0.0507

Education unknown -0.0045 - 0.0023 -0.0139 -0.0022 -0.0139Education lower/intermediate secondary 0.0048 - 0.0085 -0.0104 0.0133 -0.0104Education higher secondary -0.0009 - 0.0046 0.0221 0.0037 0.0221Education tertiary 0.007 - 0.0083 0.0617 0.0153 0.0617

Nationality (German) 0.0098 - 0.0256 0.049 0.0353 0.049

Employed: 0.0068 - 0.0272 0.0813 0.034 0.0813Regular employment 0.0036 - 0.027 0.0922 0.0306 0.0922Marginal employment 0.0064 - -0.0023 -0.0103 0.0041 -0.0103

Mean income (Eur) 31.7635 - -16.5424 401.4651 15.2211 401.4651Low income 0.0008 - 0.0052 -0.0788 0.0060 -0.0788Middle income -0.0121 - 0.0095 -0.0234 -0.0025 -0.0234High income 0.0112 - -0.0147 0.1023 -0.0035 0.1023

Current receipt UB II -0.0229 - 0.024 -0.0534 0.0011 -0.0534Past receipt of UB II -0.0414 - 0.0209 -0.1015 -0.0205 -0.1015

Table 3: Noncontact bias, refusal bias and nonresponse bias



Appendix: B - Total Bias



Rel.nonresponse

bias

Rel.measurementerrorbias

Rel.totalbias

Variable

CATI

Web

CATI

Web

CATI

Web

Female

9.06

3.90

-0.48

-0.89

8.54

2.97

(5.56;12.7)

(-1.68;9.44)

(-1.02;0.08)

(-2.11;0.18)

(3.77;13.34)

(-3.27;9.34)

Meanage(years)

1.72

-4.97

0.02

-17.47

1.73

-21.57

(0.63;2.82)

(-6.59;-3.4)

(-0.04;0.07)

(-53.76;0.57)

(0.32;3.12)

(-56.16;-3.49)

Age≤

30

-4.39

7.078

-0.23

0-4.61

7.08

(-11.62;2.6)

(-5.69;20.06)

(-0.74;0)

(-2.07;2.07)

(-13.39;4.54)

(-7.45;21.93)

Age30-39

-13.45

23.50

-0.43

-0.33

-13.82

23.09

(-19.61;-7.28)

(12.42;34.75)

(-1.47;0.63)

(-1.5

;0.71)

(-21.5

;-5.93)

(10.33;36.07)

Age40-49

4.13

0.09

0.85

1.09

5.01

1.18

(-1.91;10.29)

(-9.48;9.6)

(-0.16;1.9)

(0;2.51)

(-3.05;13.28)

(-9.32;11.96)

Age50-59

11.80

-6.87

-0.55

-0.98

11.18

-7.78

(4.98;18.87)

(-17.87;3.83)

(-1.67;0.54)

(-3.02;0.91)

(2.04;20.84)

(-19.56;4.22)

Age≥

60

3.45

-33.07

0.27

03.73

-33.07

(-4.85;11.73)

(-44.17;-21.63)

(-0.85;1.47)

(-3.48;3.67)

(-6.56;15.44)

(-44.52;-20.64)

Employed:

5.82

87.54

11.32

12.93

17.8

28.48

(2.93;8.7)

(62.24;113.77)

(8.48;14.33)

(8.98;17.25)

(14.16;21.67)

(23.81;33.11)

Regularemployment

6.71

19.79

8.65

3.48

15.94

23.95

(2.8

;10.58)

(13.75;25.76)

(5.87;11.55)

(-0.16;7.14)

(10.72;20.9)

(17.11;30.58)

Marginalemployment

4.24

-10.39

-28.63

-25

-25.61

-32.79

(-6.01;14.95)

(-26.58;6.34)

(-37.65;-19.06)

(-38.54;-9.52)

(-36.74;-13.45)

(-47.49;-16.63)

Meanincome(Eur)

0.77

20.01

-2.17

10.36

-1.42

32.45

(-2.56;4.05)

(14.4

;25.47)

(-6.14;2.11)

(1.08;21.6)

(-6.61;4.02)

(20.7

;47.23)

Lowincome

1.72

-23.18

13.37

7.11

15.15

-17.91

(-4.26;7.67)

(-31.64;-14.16)

(6.66;20.78)

(-3.95;20.24)

(6.46;23.81)

(-27.27;-8.25)

Middleincome

-0.75

-6.87

4.02

7.74

3.41

0.49

(-6.87;5.31)

(-16.01;2.6)

(-3.66;12.09)

(-2.32;19.21)

(-4.93;12.11)

(-9.49;10.44)

Highincome

-1.12

32.09

-19.74

-10.26

-20.64

18.54

(-7.5

;5.31)

(21.65;43.07)

(-25.83;-13.27)

(-17.47;-2.93)

(-28.75;-12.44)

(7.73;30.12)

Past

receiptofUBII

-5.91

-37.88

-14.78

17.78

-19.81

-26.84

(-10.6

;-1.33)

(-45.68;-29.92)

(-17.77;-11.58)

(8.12;28.49)

(-25.42;-13.99)

(-35.8

;-17.48)

Table4:

Relativenonresponse

bias,relative

measurementerrorbiasandrelative

totalbiasincludingcon�dence

intervals(0.025,0.975)



Nonresponse bias Measurement error bias Total biasVariable CATI Web CATI Web CATI Web

Female 0.0437 0.0194 -0.0025 -0.0046 0.0412 0.0148

Mean age (years) 0.7492 -2.1944 0.0083 -7.3355 0.7575 -9.5299Age ≤ 30 -0.0083 0.0118 -0.0004 0 -0.0087 0.0118Age 30-39 -0.0304 0.0526 -0.0008 -0.0009 -0.0312 0.0517Age 40-49 0.0097 0.0002 0.0021 0.0028 0.0118 0.0030Age 50-59 0.0238 -0.0139 -0.0013 -0.0018 0.0226 -0.0158Age ≥ 60 0.0051 -0.0507 0.0004 0 0.0055 -0.0507

Employed: 0.034 0.0813 0.07 0.0869 0.104 0.1682Regular employment 0.0306 0.0922 0.0421 0.0194 0.0727 0.1116Marginally employment 0.0041 -0.0103 -0.0288 -0.0222 -0.0247 -0.0325

Mean income (Eur) 15.2211 401.4651 -43.4491 249.4429 -28.228 650.9081Low income 0.006 -0.0788 0.0474 0.0185 0.0529 -0.0608Middle income -0.0025 -0.0234 0.0135 0.0247 0.0116 0.0017High income -0.0035 0.1023 -0.0609 -0.0432 -0.0644 0.0591

Past receipt of UB II -0.0205 -0.1015 -0.0483 0.0296 -0.0689 -0.0719

Table 5: Nonresponse bias, measurement error bias and total bias



Appendix: C - Comparison of Densities

20 40 60 80 100 120

0.00

00.

005

0.01

00.

015

0.02

00.

025

0.03

0

Age Distribution Web

Age

Den

sity

Administrative DataSurvey Data

20 30 40 50 60 70 80

0.00

00.

005

0.01

00.

015

0.02

00.

025

Age Distribution Cati

Age

Den

sity


Figure 5: Comparison of distributions of age across modes

The distributions show no signi�cant di�erence in the densities of administrative and survey

data. For presentation purposes one outlier (age = - 7987) has been deleted for the graph for

the web survey. However, results are robust given the inclusion of this far outlier in the test.

0 5000 10000 15000 20000

0.00

000

0.00

010

0.00

020

Income Distribution Web

Income in Euro

Den

sity


0 5000 10000 15000 20000

0.00

000

0.00

010

0.00

020

0.00

030

Income Distribution Cati

Income in Euro

Den

sity


Figure 6: Comparison of distributions of income across modes

Income densities are signi�cantly di�erent from each other in both modes (web D=.0721,

p<.05; CATI D=.062, p<.01). For both modes we observe a heap at about 6,000 Euro (due to

top-coding in the administrative data). Excluding all cases above this threshold still leads to

signi�cant di�erences between the densities that is driven by reporting error.


the e ect of survey mode on nonresponse and measurement error bias…€¦ · · 2013-05-05a...

Documents