1

Unobserved common causes of measurement and nonresponse error on the

2008 ANES Panel Survey

International Total Survey Error Workshop Stowe, VT - June 13-16, 2010

Caroline Roberts – University of Lausanne, CHPatrick Sturgis – University of Southampton, UK

Nick Allum – University of Essex, UK

Overview

• Background and motivation• Objectives of this study• Data and sample• Methods• Level of effort & response propensity analysis• Structural Equation Modeling• Provisional conclusions and discussion points

Background

• Several theoretical models specify ways measurement error and nonresponse bias might relate (see Olson 2007)

• ‘Common cause’ model: variables influencing response propensity also influence response accuracy

• Possibility to test the model restricted by data availability

• Focus on individual items does not address error from suboptimal response strategies

Motivation

• To what extent do common causes influence both types of error? – The role of motivation and ability

• Our approach uses:– Panel data – to investigate a range of candidate common

causes– Structural Equation Models to quantify the unobserved

component

• Theoretical and practical interest

Objectives

Two elements:1. Comparison of data quality based on respondent

‘cooperativeness’ in the panel: do the least cooperative differ from the most?

2. Analysis of common causes of response propensity and measurement error using SEM: what is the extent and magnitude of the unobserved component?

Data• 2008-2009 ANES Internet Panel Survey• Recruited by RDD telephone survey• Non-internet households got MSN Web-TV • 21 monthly Internet surveys: $10 each• Fieldwork by Knowledge Networks • Advance release data file (June 2009) include

• recruitment data (including CATI paradata)• core profile survey• plus 6 ANES waves (Jan, Feb, Jun, Sep, Oct, & Nov 2008)

DeBell, Krosnick, Lupia & Roberts, 2009

Sample and Fieldwork• Probability sample of US citizens aged 18+• Data from 1 of 2 recruitment cohorts• 12,809 landline numbers; 2,371 completed

recruitment (18.5%)• 4 month fieldwork – up to 50 call attempts• 2 protocol changes –

• Refusal conversion by NORC• Internet-only recruitment for 50+ calls

• AAPOR1 = 26% AAPOR3 = 42%• 1,738 completed recruitment + at least 1 ANES wave

Panel retentionNumber Percent of total

Completed interviews 2,371 100.0

Standard telephone 2,222 93.7

Refusal conversion 85 3.6

Internet 64 2.7

Completed 1+ ANES waves 1738 73.3

Profile 1545 65.2 (88.9)

Wave 1 (Jan 08) 1577 66.5 (90.7)

Wave 2 (Feb 08) 1438 60.6 (82.7)

Wave 6 (Jun 08) 1406 59.3 (80.9)

Wave 9 (Sep 08) 1466 61.8 (84.3)

Wave 10 (Oct 08) 1487 62.7 (85.6)

Wave 11 (Nov 08) 1482 62.5 (85.3)

Cooperativeness• 3 indicators of recruitment effort:

1. Number of calls to a complete interview (1-5 vs. 6 or more)

2. Whether respondent or household member refused to participate during call attempts (refused once or more vs. never refused)

3. Respondent recruited after protocol change (by internet or refusal conversion vs. by standard telephone)

• Actual response propensity• Differences in sample composition, responsiveness,

key survey estimates, data quality

Data quality• Indicators of survey satisficing (Krosnick, 1991)

• Item non-response (wave 1 only)• Non-differentiation of items with same response scale• Preference for midpoints in branched questions

• Item sets repeated across several ANES waves:• Condition of the country (5 pts); candidate liking,

attitudes to groups, policy attitudes, candidate policy positions (branched 7-pt scales)

• Validity checks • Consistency and accuracy of reports – e.g. voting (but see

Berent et al. 2010)

Results

• Few significant differences in refusal and protocol change comparisons

• But respondents recruited after 6+ calls are:– Younger and more likely to be Black, non-Hispanic– less likely to have Internet access– Less likely to be Republican and conservative– Slightly more likely to satisfice

• And reluctance at recruitment leads to lower cooperation in panel

Responsiveness

1-5 calls(n=1222)

6+ calls(n=516)

No refusal

(n=1401)Refusal (n=337)

Standard recruit

(n=1641)

Protocol change (n=97)

No. of waves started (mean)

6.34(.04)

5.85 ***(.07)

6.25 (.03)

5.95*** (.08)

6.26 (.03)

5.11 ***(.14)

No. of complete waves (mean)

6.14(.04)

5.61***(.07)

6.05(.04)

5.73**(.09)

6.05(.04)

4.82***(.15)

Completed all waves %

62.5 (1.38)

44.2*** (2.19)

59.5 (1.31)

47.2*** (2.72)

60.3 (1.20)

3.1***(1.77)

Missed 1 or more waves %

23.4 (1.21)

39.5*** (2.15)

26.1 (1.17)

36.8*** (2.63)

25.2 (1.07)

79.4*** (4.13)

Dropped out % 14.1(1.00)

16.3 (1.63)

14.4 (0.94)

16.0 (2.00)

14.6 (0.87)

17.5 (3.88)

SociodemographicsLess than 6

surveys completed

4-5 surveysCompleted

6-7 surveys completed

Male % 46.6 * 45.9 41.7

18-39 % 33.8*** 34.6*** 21.2

40-59 % 39.9** 39.7* 47.4

60+ % 26.4* 25.7* 31.4

White, nH % 76.5*** 77.4*** 87.6

Black, nH % 13.5*** 13.4** 7.0

Hispanic % 7.4** 7.2* 4.1

Other, nH % 6.5** 5.1 3.6

SociodemographicsLess than 6

surveys completed

4-5 surveysCompleted

6-7 surveys completed

< High Sch % 7.4** 6.5* 3.8

High Sch % 19.2** 19.5* 13.8

College % 35.9 38.0 38.5

Bachelor % 20.3 23.3 23.2

Graduate % 14.4** 12.3** 20.4

Northeast 17.0 15.4 15.6

Midwest 23.7* 28.4 28.4

South 32.5 33.2 31.7

West 19.0* 17.8* 24.1

Recruitment Variables

Less than 6 surveys

completed

4-5 surveysCompleted

6-7 surveys completed

Internet access % 83.2** 83.9* 88.0

Interest in politics %★ 37.9† 38.7 42.2

Interest in computers★ 41.0 39.0 41.4

Number of opinions★★ 31.4 31.8 29.0

Important to obey authority★

70.6 69.9 70.4

Wave 1 variablesLess than 6

surveys completed

4-5 surveysCompleted

6-7 surveys completed

Democrat 28.3* 29.8 32.7

Independent 22.9*** 24.0** 31.9

Republican 17.4*** 19.9*** 33.6

Liberal 22.7*** 24.0** 32.7

Moderate 18.5 20.2 20.5

Conservative 27.2*** 29.8*** 45.0

Interest in pol 33.6*** 38.4*** 55.7

Voted in 2008 75.1*** 77.0*** 88.4

Voted Obama 28.1*** 34.9** 43.5

W1 attitudes

Data qualityLess than 6

surveys completed

4-5 surveysCompleted

6-7 surveys completed

W1 MP (mean) .47* .48* .45

W1 ND (mean) .54* .54* .53

W2 MP (mean) .60* .61* .57

W2 ND (mean) .64* .64* .61

W6 MP (mean) .34** .33* .30

W6 ND (mean) .46* .46† .44

W9 MP (mean) .32* .32** .28

W9 ND (mean) .47* .47 .45

W10 MP (mean) .23 .23 .22

W10 ND (mean) .35 .36 .35

W11 MP (mean) .26† .27 .23

W11 ND (mean) .49* .49 .47

Summary• Level of effort analysis:

– Small differences between respondents as a function of ‘effort’ required to recruit them

– Significant differences in their cooperativeness at later panel waves

– Significant differences in demographics, on key survey estimates, and on satisficing between more and less cooperative panel recruits

– A few differences on substantive items used in satisficing indicators, but not many

Common causes• Ability:

– Education– Computer/Internet literacy

• R required MSN-TV device• Motivation:

– Recruitment difficulty– Interest in computers– Interest in politics

• Demographic characteristics – sex, age, race & ethnicity

Satisficing

Recruitment difficulty

Ability

Motivation

Demographics

• N of refusals • N of calls to complete

• Education

• Interest in politics• Interest in computers

• Sex• Age• Race• Web access

Response

• N of panels started

• Non-differentiation• Use of midpoints

Common cause?

Correlated residual

SEM EstimatesNo covariates +Recruitment +Ability +Motivation +Demographics

Resp Satis Resp Satis Resp Satis Resp Satis Resp Satis

Refusals -.11* .03 -.10 .02 -.10 .05 -.11* .17

Calls -.04*** .12 -.04** .03 -.04** .03 -.03** .023

Postgrad Degree .44** -3.0** .37** -1.90** .26* -1.66

College Degree .42** -2.4** .35** -1.60** .22* -.47

Some college .34** -1.8** .30** -.57 .27** -1.40**

Interest politics .37** -7.78** .24 -7.10**

Interest computers -.07 1.38** .04 1.03*

Female -.13* -1.05**

Age .01** -.01

White .40** -1.82**

Web TV -.29** .50

Residual correlation -.09** -.08** -.05 -.01 .02

RMSEA<.05; CFI>.95

Summary 2• SEM:

– Very weak correlation between satisficing and propensity to respond – a ‘reassuring’ result?

– Recruitment difficulty predicts response propensity but not satisficing

– Motivation variables better predict satisficing; ability better predicts response. Both together can jointly account for the weak correlation between propensity to respond and satisficing.

Discussion points

• Limitations: – absence of external records– advance release data– specification of SEM

• Can we improve measures of responsiveness and satisficing (including choice of item sets)?

• How can we best utilize the strengths and compensate for the limitations of the panel design

Thank you

caroline.roberts@unil.ch

Sample compositionCharacteristic Percent in sample

analyzed (n = 1738)Percent in population

(CPS, March 2008)

Male 43.2 47.8

Northeast 16.0 18.5

Midwest 27.2 23.1

South 31.9 36.7

West 22.7 21.8

18-29 9.5 21.3

30-39 15.0 16.5

40-49 21.1 19.5

50-59 24.3 18.3

60-69 18.5 12.1

70 or older 11.6 12.2

Sample compositionCharacteristic Percent in sample

analyzed (n = 1738)Percent in population

(CPS, March 2008)

Race/ ethnicity

White, non-Hispanic 81.6 73.9

Black, non-Hispanic 8.8 11.7

Hispanic 5.6 9.2

Other, non-Hispanic 4.0 5.2

Educational attainment

< High school diploma 4.8 11.9

High school diploma 15.4 31.4

Some college 38.2 29.2

Bachelor’s degree 22.7 18.4

Graduate degree 19.0 9.2

Sociodemographics

1-5 calls(n=1222)

6+ calls(n=516)

No refusal

(n=1401)Refusal (n=337)

Standard recruit

(n=1641)

Protocol change (n=97)

Male % 43.3 43.0 42.3 47.2 43.1 45.4

18-39 % 21.4 32.0*** 24.1 26.4 23.6 39.2**

40-59 % 45.8 44.4 45.6 44.5 45.7 40.2

60+ % 32.8 23.6*** 30.3 29.1 30.7 20.6*

White, nH % 83.2 77.9** 81.2 83.4 81.4 85.6

Black, nH % 7.4 12.0** 9.1 7.4 9.0 5.2

Hispanic % 5.2 6.4 5.5 5.9 5.5 6.2

Other, nH % 4.1 3.7 4.1 3.3 4.0 3.1

Sociodemographics

1-5 calls(n=1222)

6+ calls(n=516)

No refusal

(n=1401)Refusal (n=337)

Standard recruit

(n=1641)

Protocol change (n=97)

< High Sch % 4.3 5.6 4.4 6.2 4.6 6.2

High Sch % 15.8 14.0 14.7 17.5 15.2 15.5

College % 37.6 38.2 37.4 39.5 37.5 42.3

Bachelor % 22.8 21.5 22.8 20.8 22.8 16.5

Graduate % 18.6 19.4 19.5 16.0 18.8 19.6

Northeast 15.7 16.7 15.5 18.1 15.9 17.5

Midwest 24.6 28.2 27.5 25.8 27.7 17.5*

South 32.6 31.7 31.9 32.0 31.2 44.3**

West 24.2 22.1 23.0 21.7 22.9 19.6

Recruitment variables

1-5 calls(n=1222)

6+ calls(n=516)

No refusal

(n=1401)Refusal (n=337)

Standard recruit

(n=1641)

Protocol change (n=97)

Internet access %

88.2 83.3** 86.7 86.9 86.8 85.6

Interest in politics %★

41.7 39.7 41.3 40.4 40.7 47.4

Interest in computers★

40.9 42.1 42.1 37.7 41.0 46.4

Number of opinions★★

30.5 27.5 29.8 28.8 29.7 27.8

Important to obey authority★

70.9 70.3 70.7 69.7 41.0 46.4

★ Very or extremely ★★About many things or just about everything

Wave 1 variablesWave 1 variables (n=1577)

1-5 calls(n=1222)

6+ calls(n=516)

No refusal

(n=1401)Refusal (n=337)

Standard recruit

(n=1641)

Protocol change (n=97)

Democrat 31.7 31.2 31.8 30.3 31.9 25.8

Independent 29.8 28.9 29.4 30.0 29.3 34.0

Republican 30.5 26.6 29.6 28.2 29.7 22.7

Liberal 30.0 30.2 30.2 29.4 30.1 28.9

Moderate 20.1 19.6 19.6 21.7 20.2 15.5

Conservative 41.9 36.6* 41.1 37.1 40.5 38.1

Interest in pol 51.1 46.9 50.2 48.4 50.0 47.4

Planned to vote 93.6 90.5* 93.1 91.1 92.8 91.9

Voted in 2008 85.4 87.0 85.7 86.6 85.9 85.1

Voted Obama 37.6 43.8* 39.3 40.4 39.5 39.2

Data quality

1-5 calls(n=1222)

6+ calls(n=516)

No refusal

(n=1401)Refusal (n=337)

Standard recruit

(n=1641)

Protocol change (n=97)

W1 INR (mean) .01 .01 .01 .02 .01 .02

W1 MP (mean) .44 .47* .45 .46 .45 .45

W1 ND (mean) .53 .54 .53 .54 .53 .53

W2 MP (mean) .57 .59 .57 .59 .58 .59

W2 ND (mean) .61 .62 .61 .62 .62 .62

W6 MP (mean) .30 .31 .30 .32 .30 .31

W6 ND (mean) .44 .47** .44 .45 .44 .46

W9 MP (mean) .28 .29 .28 .31 .29 .30

W9 ND (mean) .46 .46 .46 .47 .46 .45

W10 MP (mean) .22 .22 .22 .23 .22 .22

W10 ND (mean) .35 .35 .35 .36 .35 .35