how to measure what we'd rather not say: list experiments and anti-immigrant sentiment mathew j...
TRANSCRIPT
1
How to measure what we'd rather not say:
List experiments and anti-immigrant sentiment
Mathew J CreightonUMass, Boston
Mathew J. Creighton - U54 SSMC - 2015
2
Motivation
In surveys, as in life, there are things that you would prefer keep to yourself.
Mathew J. Creighton - U54 SSMC - 2015
Mathew J. Creighton - U54 SSMC - 2015
3
Objectives
What is the list experiment?
How to apply it to measure… …implicit/covert opposition. …social desirability bias. …outcomes that are not dichotomous. …longitudinal trends in social desirability bias.
Throughout, we will explore some (many) drawbacks to the list experiment.
4
Examples
Four examples of applications of the list experiment will be explored
Example 1: Does religion shape our attitudes toward immigrants in the US?
Example 2: Has the US become more hostile toward immigration after the economic crisis?
Example 3: Is opposition to Muslim immigrants masked in the Netherlands?
Example 4: Are Muslim immigrants subject to greater opposition than other immigrant groups in the UK?
Mathew J. Creighton - U54 SSMC - 2015
5
Examples: The 1st and 2nd are a reasonable introduction to the technique
Four examples of applications of the list experiment to measure (in)tolerance
Example 1: Does religion shape our attitudes toward immigrants in the US?
Example 2: Has the US become more hostile toward immigration after the economic crisis?
Example 3: Is opposition to Muslim immigrants masked in the Netherlands?
Example 4: Are Muslim immigrants subject to greater opposition than other immigrant groups in the UK?
Mathew J. Creighton - U54 SSMC - 2015
Mathew J. Creighton - U54 SSMC - 2015
6
Objectives
What is the list experiment?
How to apply it to measure… …implicit/covert opposition. …social desirability bias. …outcomes that are not dichotomous. …longitudinal trends in social desirability bias.
Throughout, we will explore some (many) drawbacks to the list experiment.
Mathew J. Creighton - U54 SSMC - 2015
7
Example 1: Social Identity Theory and Citizenship for Muslim Immigrants in the US
Social Identity Theory (SIT)*: Individuals express greater affinity for members of their own “group”.
Ingroup bias (Tajfel 1970, Tajfel et al. 1971; Tajfel and Turner 1986)
Outgroup hostility
*For thorough review of the literature see Brown (2000)
Mathew J. Creighton - U54 SSMC - 2015
8
Social identity theory (SIT) applied to immigrant reception and religion
Religion is a group identity Co-religious immigrants benefit from ingroup
bias Immigrants from other religions do not benefit
from ingroup bias
Example 1: Social Identity Theory, religion and immigration
Mathew J. Creighton - U54 SSMC - 2015
9
Example 1: Hypotheses
H1: Explicit opposition to citizenship for legal Muslim immigrants is greater than explicit opposition to citizenship for legal Christian immigrants.
H2: Implicit opposition to citizenship for legal Muslim immigrants is no greater than implicit opposition to citizenship for legal Christian immigrants.
Mathew J. Creighton - U54 SSMC - 2015
10
Social desirability bias
Affirmative action (Kuklinski et al. 1997b)
Same-sex marriage (Janus 2011)
Closing the US border (Janus 2011)
Immigration policy preferences (Knoll 2013)
Nativism (Knoll 2013)
Religious service attendance (Brenner 2011, 2012a, 2012b)
Race attitudes (Kuklinsky et al. 1997)
Mathew J. Creighton - U54 SSMC - 2015
11
Time-Sharing Experiments in Social Science (TESS) and Knowledge Networks (KN)
Random-digit dialing (RDD) and Address-based sampling (ABS)
Provision of free computers or WebTV’s
Sample of adults 18+
Representative of 97% of U.S. households
Response rate of 66.6%
Mathew J. Creighton - U54 SSMC - 2015
12
Control:Direct
QuestionsN = 788
Treatment 1:Opposition to Citizenship for Legal Muslim Immigrants
n = 804
Treatment 2:Opposition to Citizenship for Legal Christian
Immigrantsn = 774
Treatment 3:Support for
Closed Bordersn = 836
Mathew J. Creighton - U54 SSMC - 2015
13
Example 1: The control (explicit opposition)
Below you will read three things that sometimes people oppose or are against. After you read all three, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.
1. the federal government increasing assistance to the poor
2. professional athletes making millions of dollars per year
3. large corporations polluting the environment
Do you support or oppose granting citizenship to a legal immigrant who is Muslim?
Do you support or oppose granting citizenship to a legal immigrant who is Christian?
Mathew J. Creighton - U54 SSMC - 2015
14
Example 1: The treatment (implicit opposition)
Below you will read four things that sometimes people oppose or are against. After you read all four, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.
1. the federal government increasing assistance to the poor
2. professional athletes making millions of dollars per year
3. large corporations polluting the environment
4.(Treatment 1)
granting citizenship to a legal immigrant who is Muslim
4.(Treatment 2)
granting citizenship to a legal immigrant who is Christian
Mathew J. Creighton - U54 SSMC - 2015
15
Standard Estimation Strategy: Implicit Opposition
Difference in Means:
Recent developments: Maximum likelihood (Imai, 2011; Graeme and Imai, 2011)
R Package list (Graeme and Imai, 2011)
Mathew J. Creighton - U54 SSMC - 2015
16
Measuring social desirability bias
Proportion opposed to when asked directly
(i.e., explicit opposition)
Proportion opposed when asked indirectly
(i.e., implicit opposition)
Mathew J. Creighton - U54 SSMC - 2015
17
Christian immigrants: Implicit vs. explicit
0.28 [±0.07]
0.17 [±0.07]
0.11 [±0.02]
Source: TESS/Knowledge Networks® 2010
Mathew J. Creighton - U54 SSMC - 2015
18
Muslim immigrants: Implicit vs. explicit
0.33 [±0.06] 0.03 [±0.08]0.30 [±0.03]
Source: TESS/Knowledge Networks® 2010
Mathew J. Creighton - U54 SSMC - 2015
19
Example 1: Findings
Opposition to citizenship for legal Muslim immigrants is not greater, only more openly expressed.
Religious ingroup membership for Christian immigrants does not always lend itself to ingroup protections. What it does offer is protection from anti-immigrant sentiment’s explicit expression.
20
Examples
Four examples of applications of the list experiment will be explored
Example 1: Does religion shape our attitudes toward immigrants in the US?
Example 2: Has the US become more hostile toward immigration after the economic crisis?
Example 3: Is opposition to Muslim immigrants masked in the Netherlands?
Example 4: Are Muslim immigrants subject to greater opposition than other immigrant groups in the UK?
Mathew J. Creighton - U54 SSMC - 2015
Mathew J. Creighton - U54 SSMC - 2015
21
Objectives
What is the list experiment?
How to apply it to measure… …implicit/covert opposition. …social desirability bias. …outcomes that are not dichotomous. …longitudinal trends in social desirability bias.
Throughout, we will explore some (many) drawbacks to the list experiment.
Mathew J. Creighton - U54 SSMC - 2015
22
Example 2: Labor Market Competition Theory, the economic Crisis and immigration
Labor Market Competition Theory (LMCT)*
“As rational actors, natives pursuing their own wellbeing develop unfavorable attitudes in order to
legitimate their social positions when competing with foreigners over jobs…, especially during times of
economic recession” (Ceobanu et al. 2010: 317)
*For thorough review of the literature see Ceobanu et al. (2010).
Mathew J. Creighton - U54 SSMC - 2015
23
Example 2: Evidence is mixed
Of 21 peer-reviewed studies over 15 years*
4 are strongly supportive 8 are weakly supportive 9 find no support whatsoever
Main evidence is derived from measures of education or unemployment
*For thorough review of recent findings see Malhorta et al. (2010).
Mathew J. Creighton - U54 SSMC - 2015
24
Mathew J. Creighton - U54 SSMC - 2015
25
Control:Direct
QuestionsN = 788
Treatment 1:Opposition to Citizenship for Legal Muslim Immigrants
n = 804
Treatment 2:Opposition to Citizenship for Legal Christian
Immigrantsn = 774
Treatment 3:Support for
Closed Bordersn = 836
Mathew J. Creighton - U54 SSMC - 2015
26
Example 2: The control (explicit opposition)
Below you will read three things that sometimes people oppose or are against. After you read all three, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.
1. the federal government increasing assistance to the poor
2. professional athletes making millions of dollars per year
3. large corporations polluting the environment
Do you support or oppose cutting off all immigration to the United States?
Mathew J. Creighton - U54 SSMC - 2015
27
Example 2: The treatment (implicit opposition)
Below you will read four things that sometimes people oppose or are against. After you read all four, just tell us HOW MANY of them you OPPOSE. We don’t want to know which ones, just HOW MANY.
1. the federal government increasing assistance to the poor
2. professional athletes making millions of dollars per year
3. large corporations polluting the environment
4.(Treatment 3)
Cutting off all immigration to the United States
Mathew J. Creighton - U54 SSMC - 2015
28
Example 2: Hypotheses
H1: After the financial crisis, opposition to immigration increased. H2: After the financial crisis, opposition to immigration increased more among the less educated than the more educated. H3: After the financial crisis, opposition to immigration increased more among the unemployed than the employed. H4: After the financial crisis, opposition to immigration increased more among those with a lower household income than those with a higher household income.
Mathew J. Creighton - U54 SSMC - 2015
29
Example 2: Hypothesis 1
H1: After the financial crisis, opposition to immigration increased. H2: After the financial crisis, opposition to immigration increased more among the less educated than the more educated. H3: After the financial crisis, opposition to immigration increased more among the unemployed than the employed. H4: After the financial crisis, opposition to immigration increased more among those with a lower household income than those with a higher household income.
Mathew J. Creighton - U54 SSMC - 2015
30
Example 2: Overall pre-crisis to post-crisis trend
SocialDesirabilityBias
SocialDesirabilityBias
60%
Mathew J. Creighton - U54 SSMC - 2015
31
Example 2: Pre-crisis to post-crisis trend
Explicit opposition Implicit opposition
32
Examples
Four examples of applications of the list experiment will be explored
Example 1: Does religion shape our attitudes toward immigrants in the US?
Example 2: Has the US become more hostile toward immigration after the economic crisis?
Example 3: Is opposition to Muslim immigrants masked in the Netherlands?
Example 4: Are Muslim immigrants subject to greater opposition than other immigrant groups in the UK?
Mathew J. Creighton - U54 SSMC - 2015
Mathew J. Creighton - U54 SSMC - 2015
33
Objectives
What is the list experiment?
How to apply it to measure… …implicit/covert opposition. …social desirability bias. …outcomes that are not dichotomous. …longitudinal trends in social desirability bias.
Throughout, we will explore some (many) drawbacks to the list experiment.
Mathew J. Creighton - U54 SSMC - 2015
34
Example 3: Converting direct questions from the ESS to a list experiment in the Netherlands
One of the primary drawbacks of a list experiment is that the response must be a binary.
Typically, direct questions offer more options and/or a neutral category
This example is an effort to convert a scaled response (i.e., some, few, many) to a binary response (i.e., none, some).
Mathew J. Creighton - U54 SSMC - 2015
35
Example 3: These are the original ESS questions (first asked in 2003)
Mathew J. Creighton - U54 SSMC - 2015
36
Example 3: The binary conversion
Mathew J. Creighton - U54 SSMC - 2015
37
Example 3: The binary conversion
Mathew J. Creighton - U54 SSMC - 2015
38
Example 3: The list experiment
Mathew J. Creighton - U54 SSMC - 2015
39
Example 3: The Data
Mathew J. Creighton - U54 SSMC - 2015
40
Control:Direct
QuestionsAnd 3-
question list
Treatment 1:ESS questions
Treatment 2:List experiment
(poor immigrants)
Treatment 3:List experiment
(same race)Treatment 4:
List experiment (different race)
Treatment 5:List experiment
(Muslim immigrants)
Mathew J. Creighton - U54 SSMC - 2015
41
Example 3: Preliminary results
*Binary conversion analysis is pending. Check in after ESRA Conference
42
Examples: The 1st and 2nd are a reasonable introduction to the technique
Four examples of applications of the list experiment to measure (in)tolerance
Example 1: Does religion shape our attitudes toward immigrants in the US?
Example 2: Has the US become more hostile toward immigration after the economic crisis?
Example 3: Is opposition to Muslim immigrants masked in the Netherlands?
Example 4: Are Muslim immigrants subject to greater opposition than other immigrant groups in the UK?
Mathew J. Creighton - U54 SSMC - 2015
Mathew J. Creighton - U54 SSMC - 2015
43
Objectives
What is the list experiment?
How to apply it to measure… …implicit/covert opposition. …social desirability bias. …outcomes that are not dichotomous. …longitudinal trends in social desirability bias.
Throughout, we will explore some (many) drawbacks to the list experiment.
Mathew J. Creighton - U54 SSMC - 2015
44
Example 4: Longitudinal approach
List experiments are almost always cross-sectional
The stability of social desirability bias over time is hard to determine.
If, for some questions, the bias is stable, corrections are easier to make.
Mathew J. Creighton - U54 SSMC - 2015
45
Example 4: Stability over time
Wave 1
A control and three treatments are asked about certain immigrant groups in the UK
Wave 2
The treatment and controls are randomized for a subset in the second wave.
T1 T2
Mathew J. Creighton - U54 SSMC - 2015
46
Example 4: The baseline list
Mathew J. Creighton - U54 SSMC - 2015
47
Example 4: The Innovation Panel
Special module of the UK Household Longitudinal Study
Intention is to test innovative experimental designs at the population level that focus on substantive questions
Our experiment is in beta testing and should enter the field soon.
Mathew J. Creighton - U54 SSMC - 2015
48
Control:Direct
Questions and control
listW1C1
Treatment 1:Caribbean immigrants
W1T1
Treatment 2:Eastern
European immigrants
W1T2
Treatment 3:Muslim
immigrantsW1T3
Wave 1:
Mathew J. Creighton - U54 SSMC - 2015
49
Control:Direct
Questions and control
listW2C1
(W1C1 + Random sub-set)
Treatment 1:Caribbean immigrants
W2T1(W1T1 + Random sub-
set)
Treatment 2:Eastern
European immigrants
W2T2(W1T2 + Random sub-
set)
Treatment 3:Muslim
immigrantsW2T3
(W1T3 + Random sub-set)
Wave 2:
Mathew J. Creighton - U54 SSMC - 2015
50
Example 4: Preliminary results
Mathew J. Creighton - U54 SSMC - 2015
51
Recap
List experiment allows total and permanent anonymity.
When compared to a direct question, social desirability bias can be directly assessed and it’s magnitude quantified.
Dichotomous response patterns are not your only option, but converting from a more complicated direct question should, at the very least, involve a conversion process in the experiment itself.
Longitudinal measures of social desirability bias can be done with repeated cross-sections or with a true longitudinal set-up. The advantages of the latter remain to be seen.
Mathew J. Creighton - U54 SSMC - 2015
52
Acknowledgements
Time-Sharing Experiments in Social Science (TESS) Jeremy Freese (Northwestern) Penny Visser (University of Chicago)
Social, Behavioral and Economic Sciences Directorate of the National Science Foundation
Mathew J. Creighton - U54 SSMC - 2015
53
Acknowledgements
LISS Panel (The Netherlands)
Innovation Panel of the UK Household Longitudinal Study
Mathew J. Creighton - U54 SSMC - 2015
54
Thank you
<there are bonus slides after this to fill time if nobody asks questions>
Mathew J. Creighton - U54 SSMC - 2015
55
Step 2: Pre-crisis to post-crisis trend: high school
SocialDesirabilityBias Social
DesirabilityBias
Mathew J. Creighton - U54 SSMC - 2015
56
Step 2: Pre-crisis to post-crisis trend: some college
SocialDesirabilityBias Social
DesirabilityBias
Mathew J. Creighton - U54 SSMC - 2015
57
Step 2: Pre-crisis to post-crisis trend: college or more
SocialDesirabilityBias Social
DesirabilityBias
Mathew J. Creighton - U54 SSMC - 2015
58
The Maximum Likelihood Estimator (Imai 2011)