Survey Experiments: Illuminating Theoretical and

Methodological Controversies

Harold Clarke

EPPS, University of Texas at Dallas

email: clarke475@msn.com

Utility of Survey Experiments

• Survey experiments – increasingly popular over past decade

• Sold as way to study causal processes that cannot be investigated with traditional cross-sectional & panel surveys

• Randomization as cure for omitted variable biases & endogeneity problems that bedevil causal inference in observational studies

• Survey experiments – external validity - sophomores in labs are not representative!

• Other problems – truncated variance, small N’s, functional form forgotten by ANOVA, model specification

Survey Experiment Vehicles• In-person probability surveys – ANES, BES, GSS• Mail-back surveys• RDD telephone surveys – impact of TESS – NSF

funds• Internet surveys – general populations - BES

Continuous Monitoring Survey (CMS), BES Rolling Campaign Panel Survey (RCPS), now ANES

• Internet surveys – specific populations – Survey Monkey, Qualtrics -> e.g., students, employees, patients, party members

• Amazon Mechanical Turk too!

Internet Survey Experiments

• Internet methodology – exciting possibilities

• Huge N’s – massive statistical power

• Inexpensive – vastly expand research opportunities – e.g., CMS

• Technical tools – video, audio, feedback to respondents, intricate randomizations

Theory & Method• Often think of experiments as theory or, at least,

hypothesis testing• Hypotheses of interest may concern measurement of

key concepts• Survey measures may be affected by context in which

questions are asked• Internal – question ordering effects• External – political/economic conditions• Gaines, Kuklinski & Quirk (2007) discuss the latter in

terms of survey experiments, but it is of general interest• Use survey experiments to investigate how external

context affects survey responses or relationships among variables – cross-level interactions

External Context Survey Experiment: Inglehart and Post-Materialism

• Testing the impact of macro-economic context on responses to Inglehart’s Materialism-Postmaterialism question battery in the Euro-Barometer surveys

• Does Inglehart’s battery really measure what he says it does?

• Or, is it contaminated by changing economic context

• Why does this matter?

Inglehart’s Value Shift Thesis & His Values Battery

Inglehart famously argued there has been a “value shift” from materialist to postmaterialism driven by rising affluence in post-war Western democracies. Everybody jumped on this bandwagon!

Four-Item Values Battery administered in bi-annual Euro-Barometer surveys

Items:• Fight Rising Prices• Maintain Order in the Nation• Give People More Say in Government• Protect Freedom of Speech

Respondents asked to select the most important, then select the next most important of the 3 remaining items.

Figure 1. Why The Battery Might Be Affected by Macro-Economic Context

0

2

4

6

8

10

12

14

0

4

8

12

16

20

24

28

74 76 78 80 82 84 86 88 90 92 94 96

Pe

r ce

nt P

er ce

nt

Unemployment

Inflation

Year

Figure 1. Inflation and Unemployment Rates,Britain January 1974 - June 1997

r = -.70

<-Thatcher Elected

Figure 2. Germany – Canonical Case

-1

0

1

2

3

4

5

6

7

8

-40

-30

-20

-10

0

10

76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92

Infla

tion

& U

ne

mp

loym

en

t - P

erc

en

tag

eB

ala

nce

Po

st-Ma

teria

lists - Ma

teria

lists - Pe

rcen

tag

e

Unemployment

Inflation

Balance Post-MaterialistsMinus Materialists

Correlation: Balance & Unemployment = +76 Balance & Inflation = -.79

Figure 2. Inflation, Unemployment & Balance of Inglehart Post-Materialistsand Materialists, Germany 1976 -1992

Values Battery Survey Experiment

• Clarke’s Chance Encounter at Zuma• Dinner With Max Kaase• What are you working on Harold?• Studying how Macro-Economic Conditions Affect

Responses to Inglehart’s Values Battery• Nice aggregate results, but need to nail it at the

individual level• Survey experiment!• Parallel work being done by Kaase in Germany• Let’s work together!

The Experiment

• National Samples of Canadian & German Electorates

• Split-Sample Design• Half-One gets traditional Values Battery; Half-

Two gets Values Battery with Unemployment item substituted for Inflation Item

• At end of Canadian survey, ask other battery – Do people change their value classification in a 15-minute survey?!

• Results?

Figure 3. Alternative Values Batteries, Responses - Canada

Figure 3. Responses to Alternative Versions of Inglehart Values Battery

18

9

52

3

17

35

25

15

7

18

0

10

20

30

40

50

60

Maintain Order More Say Jobs/Prices Free Speech MultipleMentions

Per

cen

t

Unemployment Version Inflation Version

Figure 4. Materialists & Post-MaterialistsRival Values Batteries - Canada

FIgure 4. Materialists and Post-Materiailists - Alternative Versions of Inglehart Values Battery

38

52

11

-27

25

58

17

-8

-40

-30

-20

-10

0

10

20

30

40

50

60

70

Materialist Mixed Post-Materialist Balance

Per

cen

t

Unemployment Version Inflation Version

Results & Implications

• Same results in Germany!

• Significant portion of the hypothesized value shift an artifact of interaction between his measure and changing macroeconomic context

• Example of Gaines’ et al. caveat in their Political Analysis article

• Simple experiments can have big payoffs

Three Other Examples of Survey Experiments

• The Dynamics of Partisanship – Does party identification have an individual dynamic? One aspect of controversy concerns question wording – A task for survey experiments!

• Does Mode Matter? - Do Traditional In-Person and Internet Election Studies Tell the Same Story?

• The Endogeneity of Ideal Points? – Leveraging the Power of the Internet With a Feedback-to-Respondent Experiment to determine if “De Gustibus” is a viable psephological anthropology.

Dynamics of Partisanship

• Longstanding Controversy!• Campbell et al. v. Fiorina et al.• Green et al.• In Canada, Richard Johnston & Andre Blais said

if only we changed the “or what?” in first ANES party id question to “or none of these” we would find:

• 1. fewer identifiers• 2. more stable identifiers

Survey Experiment on Stability of Party Identification in Canada

• Split Half Samples in 2000-2002 Political Support in Canada Panel Surveys

• One Half Gets Traditional Wording; Other Half Gets Revised Wording

• Results: • Slightly fewer identifiers with revised wording• Rates of partisan stability almost exactly the same • Lots of partisan instability in both versions• Survey experiment results reinforce MMLC analyses of

US, UK, Canadian and German multi-wave panel data Clarke & McCutcheon POQ 2009 and Stegmueller, Neundorf and Scotto, POQ 2011.

Figure 8. Dynamics of Party Identification, 2000-2002 PSC Panel: Stable Identifiers

Traditional and Revised Question Wordings

59.6 60.259.4 58.6

0

10

20

30

40

50

60

70

80

90

100

1st Question Both Questions

Perc

ent

Traditional Revised

Does Mode Matter? Are Ideal Points Endogenous?

• Mode – see modecomp1.ppt

• Ideal Points – see endogjepoprev.pdf

Conclusions I

• Survey experiments are useful tools!

• Survey experiments can address big theoretical & methodological controversies

• Internet surveys put survey experiments “in reach” for many researchers

• CCAP, CCES, CMS – example vehicles

• Qualtrics – sophisticated platform – many universities subscribe – no cost

Conclusions II• Survey experiments not a panacea• Example of ongoing controversies over

implications of endogenity of economic evaluations in party support and voting models

• Never resolve controversy with traditional cross-sectional or panel survey designs!

• Survey experiments very unlikely to simulate real economic conditions

• Way forward - use CMS monthly surveys 1997- 2010 & VEC modeling – see Snark Paper

• Data Driven Solutions • Not Necessarily Expensive