adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with...

Adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with

other approaches in ESS

Jaak Billiet: CeSO - K.U. Leuven Hideko Matsuo: CeSO – K.U. Leuven

The European Social Survey Round 4 launching conference ‘Poland and Europe: continuation and change’.

Institute of Philosophy and Sociology Polish Academy of Sciences, Warsaw 13 Jan 2010

Outline

Very short introduction

Short overview: Approaches to the assessment of bias applied in ESS (Billiet, Matsuo, Beullens & Vehovar, Research & Methods. ASK, vol 18 (1, 2009), pp. 3-43).

The surveys among nonrespondents (post R3): what, how, when, with what…?

Main results of NRS in PL and NO (comparison of results and focus on adjusting samples)

Pros and cons of NRS compared with other 3 approaches

1. Introduction Analysis of nr bias still needed:

WHY? Still large differences in NR rates based on CF R4

Short overview of approaches to the assessment of bias applied in ESS (Billiet, Matsuo, Beullens & Vehovar, Research & Methods. ASK. vol 18 (1, 2009), pp. 3-43).

In all rounds (R1, R2, R3, R4…..)

1. Bias as deviation between obtained sample and population (or ‘Golden standard’ survey) = post-stratification and evaluations of samples before and after weighting

2. Bias as difference between cooperative and converted refusals collected via refusal conversion = comparison of cooperative with reluctant respondents (converted refusals)

3. Bias as difference in ‘observable’ data among all sampling units (collected in contact forms)= sample based comparison between all respondents and all nonrespondents

2. Short overview (1)

In context of R3

4. Bias as difference between respondents and non-respondents collected via post hoc nonresponse survey= surveys among nonrespondents after R3 in PL, NO and CH (real NRS)

in BE (at moment of refusal only among refusals = Doorstep Questions Survey)

Short overview (2)

3. Survey among nonrespondents (1)

New survey among refusals with very small & easy questionnaire (some crucial variables) (Voogt, 2004; Saris)

Implemented in ESS Round 3 : 4 participating countries- Full mail survey (15 questions) months after main survey in NO (medium rr), CH (low rr) & PL (high rr)

- At moment of refusal 7 crucial questions in BE (7 questions)

Response rates

BE (44.7% = 303) response among refusals

NO (30.3% = 342) response among noncontacts & refusals

PL (23.2% = 192)

CH (52.9% = 771)

(cooperative much higher response)

Survey among nonrespondents (2)

1. The questions asked

Key questions procedure (Pedaksi approach)

Short 7 question module (+ at door): work situation, highest level of education, # of members in household, frequency of social activities, feeling (un)safe, interest in politics, attitude towards surveys

Normal 16 questions module: same as short + gender, year of birth, TV watching, voluntary work, trust in people, satisfied with democracy, trust in politics, immigration good/worse for country, (+ reasons for refusal (closed) in one subgroup)

Target group

Timing Mode Use of Incen-tives

Type of questio-naaire

Response

Rates (R/NR)

Sample size

BE ESS3

refusers

Same as ESS

PAPI at door

NO 1 short DQS_R: 44.7%

303

CH ESS3_R

& ESS3_NR

After ESS

Mail/

Web/

CATI

10 Swiss FR.

2 short & long

NRS3_R: 84%

NRS3_NR: 51.8%

1023

NO ESS3_R

& ESS3_NR

After ESS

Mail/

Web/

CATI

NO 1 long NRS3_R: 60.79%

NRS3_NR: 30.25%

487

PL ESS3_R

& ESS3_NR

After ESS

Mail Notepad 2 short & long+

NRS3_R: 59.04%

NRS3_NR: 23.24%

1208


2. Overview of the sample design


3. Kinds of respondents in NRS decisions to take in view of computing propensity scores for weighting the sample

NRS/(cooperative vs. nrs)

NRS/(cooperative vs. main)

(NRS+reluctant) vs (cooperative (nrs or main?))

NRS/cooperative vs reluctant/cooperative

see Figure next slide

ESS Cooperative Respondent(ESS3_Rco)

ESS Reluctant Respondent(ESS3_Rrel)

ESS Non-Respondent (ESS3_NR)

Kinds of respondents in data analyses [NO, CH & PL]

NRS Cooperative Respondent(NRS3_Rco)

NRS Reluctant Respondent(NRS3_Rrel)

ESS

NRSNRS Non-

Respondent(NRS3_NR)


Method used for adjusting the sample for nonresponse bias

1. Identify survey response differences on key explanatory variables between types of respondent (‘nonrespondent vs. cooperative respondent’).

2. Study net effects of key explanatory variables on response probabilities via logistic regression model (dependent variable: prob ratio’s ‘nonrespondent/cooperative’).

3. Obtain propensity scores on all cases on non-response probabilities via logistic regression model (dependent variable: prob ratio’s ‘cooperative/nonrespondent’).

log ( ) / (1 ( )) ' ( )e e f x x x


4. Transform propensity scores into weights via stratification

method (Rosenbaum & Rubin 1984; Little 1986; Lee &

Vaillant 2008):

Form 10 strata with equal number of cases after sorting on ps;

Assign each sample unit into correct corresponding sub-strata

Weight = expected probability/observed probability of the

coop. respondent (or nonrespondent) in the corresponding

sub-strata. 5.


5. Evaluate effects of propensity weighting via two main

criteria:

1. Tests between unweighted & weighted sample on cooperative

respondents (NRS3_Rco & ESS3_Rco).

1b. In case of significant differences: test differences between

parameters of relevant substantive explanatory models

2. Study differences in distributions on key questions between

types of respondents (NRS3_Rco vs NRS3_NR or ESS3_Rco

vs. NRS3_NR).


4. Main results in NO and PL (1)1. Differences between ESS cooperative and NRS !nonrespondents*

1.

* Only single ESS cooperative respondents (not ‘double’ respondents). All tests: ESS resp = expected freq

!

Main results in NO and PL (2)

…differences in distributions

!


2. Logistic regression parameters nonresp/cooperative

*NRS res are final ESS nonrespondents

NO: NRS res / ESS res (226 vs. 1616)

PL: NRSres / ESS res ( 156 vs. 1434 )

Odds ratio SE Odds ratio SE

Educational level (numeric: 0-8) (NO) Educational level (PL)

ISCED 2 ISCED 3 & 4 ISCED 5 & 6

Reference: ISCED 0 & 1

0.851** 0.045

0.894 1.365*

1.772**

0.146 0.134 0.184

Work status employed

Reference: unemployed

0.742**

0.091

Neighbourhood security after darkness Safe

Unsafe & very unsafe Ref: very safe

1.335* 1.344*

0.129 0.143

Involved in charity organization

(numeric: 0-6: >= once a week – never)

0.996

0.047


(continued) Logistic regression parameters nonresp/cooperative

NO: NRS res / ESS res (226 vs. 1616)

PL: NRS res / ESS res ( 156 vs. 1434 )

Odds ratio SE Odds ratio

Participation in social activities Less than most

About the same Much more than most

Ref: much less than most

0.829 0.887

0.520**

0.156 0.118 0.194

0.548***

1.287

0.130 0.158

TV watching time per day (numeric: 0-7: no time – 3+hrs per day)

1.072

0.042

Political interest Hardly interested

Not at all interested Ref: very & quite interested

0.936 1.334

0.107 0.154

0.895

0.087

How satisfied with democracy works (numeric 0-10: ex. dissatisfied - ex.satisfied)

0.916*

0.037

Immig. make country worse/better place to live (numeric 0-10: worse place – better place)

0.916*

0.036

R²=0.052; H & L= 9.060

R²=0.042; H & L=3.835

***p-value<0.0001; **p-value<0.01; p < 0.05; H&L stands for Hosmer and Lemeshow


?

Main net effects on probability ratio coop resp / nonresp (inversed parameters!) In Norway: probability of response INCREASES if • Higher educated• Participate more in social activities then most (subjective…)• More satisfied with democracy• Positive attitude towards ‘consequences’ of immigration

In Poland: probability of nonresponse INCREASES if• Higher educated!!!• Unemployed• Feel safe• Participate less in social activities than most!• (political interested?!!)


3. Evaluation of the propensity weights

First approach A: is the adjusted sample (weighted) of cooperative ESS respondents significant different from the original sample?

if yes: we may conclude that the adjustment had effect on the sample estimates

conclusion: no significant differences at all example: variable with largest differences = education


Differences between original sample and adjusted sample even smaller in PL

Not necessary to test a substantive regression model since the univariate distributions do not differ (first approach B)

This is nonetheless checked for model with “consequences of immigration” as relevant dependent variable” and number of predictor variables: age, TV watching, involvement in charity org, trust in politics, social trust, and two value orientations (conservation, self-transcendence)

R² = 0,26 in both models (not weighted & weighted)

all predictors contribute significantly to variance of dept. var

BUT: no differences at all between the two models

Conclusion = was ps weighting useless? Let us see the second approach


Second approach: do the initial significant differences of belonging to a response category of all key questions between ESS respondents and nonrespondents (NRS res) in first table disappear after adjusting the sample of ESS cooperative respondents?

in other words, did we move from NMAR to MAR

let us see:


Norway sample (Chisq values or t-values; p-values)

largely successful: all differences disappeared except political interest

(2) NRS res. vs ESS res.

Unweighted prob. Weighted prob.

Education level (df=2) 40.552 <.0001 1.813 0.404

Work status (df=1) 11.594 0.0007 0.470 0.493

Political interest (df=3) 33.014 <.0001 13.247 0.010

Participation in social activities (df=4) 48.105 <.0001 2.301 0.681

T-value prob. T-value prob.

How satisfied democracy (df=1864) 5.31 <.0001 1.01 0.313

Imm. make country worse/better place (df=1871) 5.17 <.0001 0.56 0.577

TV watching time per day (df=1878) -4.38 <.0001 -0.81 0.420

Involved in work for voluntary & charity org.

(df=1885)

-2.69 0.007 -0.69 0.492

* Only key questions with significant differences in distribution (p<.05) in unweighted sample (Table 1) are shown.


Poland: sample (Chisq values and p-values)

Not completely successful since still sign differences between NRS and ESS for two variables (political interest and social participation)

(2) NRS res. vs ESS res.

Unweighted prob. Weighted prob.

Educational level (df = 3) 14.481 0.002 0.838 0.840

Work status (df = 1) 4.658 0.031 0.340 0.560

Neighbourhood security (AESFDRK) (df=3) 14.295 0.003 2.174 0.537

Political interest (POLINTR) (df = 3) 22.141 <.0001 26.759 <.0001

Social participation (SCLACT) (df = 4) 60.838 <.0001 28.690 <.0001 * Only key questions with significant differences in distribution (p<.05) in unweighted sample (Table 7) are shown. ** Chi² is computed for partial cross-tables: (1) NRS respondent vs double respondent and (2) NRS respondent vs ESS respondent.

5. Pros and cons of NRS compared with 3 other approaches

Approach to study of nonresponse bias in cross-nation research Post-

stratification Comparison coop – reluct

Info from observ data

NRS

Possible in ESS yes yes yes yes

Adjustment of all samples

yes +/- yes +/-

Weakness 1 Small effect Small effect but info on all

variables

Small effect Small effect but info on more relevant vars

Weakness 2 No info on target vars

Differences btw countries

Measurement error in obs

NRS resp differ from ESS resp

Problem Error in Gold standard

(education…)

Final refusals differ from

converted ref.

Only small # of vars with

low cov

mode effects possible

Additional information population Some refusals All samp units Selection of #

How to adjust Ps or prop weights

Not useful Prop weights but low effect

Kinds of prop. weights

Additional costs Low Rather high Rather low Rather high

Conclusions

Future:

- Other methods (contacting sequences using contact forms data) = expect low effect (result of some studies, see Blom)

- More model based method: crucial is what additional information can be used

- Combining different methods

- Info in all methods = view on sensitive variables

- Finally: low effect may mean LOW BIAS

adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with...

Documents

frame of addresses

frame of households

business addresses

addresses inireland

residential addresses

address database

random route procedureie

ess billiet