SSRC Eurasia Quantitative Methods WebinarCultural Context and Measurement Validity in Comparative Survey Research

Professor Jane Zavisca University of Arizona

April 12, 2013 janez@u.arizona.edu

Core dilemmas In comparing groups, want to be sure observed

differences (or similarities) are substantive, not artifactual

Simultaneous need for: Identities: universal measures with comparable

meanings across contexts Equivalents: Particular measures that capture

same concepts across contexts In cross-national surveys, often have identities –

but are they equivalent?Bollen, Kenneth A., Barbara Entwisle, and Arthur S. Alderson. 1993. “Macrocomparative Research Methods.” Annual Review of Sociology 19 (January 1): 321–351. doi:10.2307/2083391.

Przeworski, Adam, and Henry Teune. 1966. “Equivalence in Cross-National Research.” Public Opinion Quarterly 30 (4) (December 21): 551–568. doi:10.1086/267455.

Heath, Anthony, Stephen Fisher, and Shawna Smith. 2005. “The Globalization of Public Opinion Research.” Annual Review of Political Science 8 (1): 297–333. doi:10.1146/annurev.polisci.8.090203.103000.

Measurement equivalence Functional equivalence: concordance of

meaning, of constructs as well as questions Measurement invariance: formal statistical

equality of parameters in measurement models

Sources of non-equivalence Errors of observation

Substantive differences in meaning: linguistic, contextual

Systematic differences in response styles: acquiescence bias, extreme response bias, social acceptability bias

Errors of non-observation non-response bias Sampling approach

Heath, Anthony, Jean Martin, and Thees Spreckelsen. 2009. “Cross-national Comparability of Survey Attitude Measures.” International Journal of Public Opinion Research 21(3): 293–315. doi:10.1093/ijpor/edp034.

Language example: happiness Poor linguistic equivalence

English: “Are you a happy person?” Russian:  ”Вы счастливый человек? / Вы

счастливчик?» Could be interpreted as “Are you a lucky person?” О счастливчик! = name of “Who Wants to Be a

Millionaire” reality show in Russian Better linguistic equivalence

English: “Are you: very happy, pretty happy, not too happy, not happy at all.”

Russian: Вы: очень счастливы, довольно счастливы, не очень счастливы, очень несчастны.»

See: RUSSET Panel Survey: www.vanderveld.nlWorld Database of Happiness: http://www1.eur.nl/fsw/happiness/index.html

Relationship between happiness & life satisfaction “Varies with the cultural and linguistic

environment in which it is studied.” Constructs vs. questions: differences in meaning

even with linguistic equivalence In Russian surveys

Low correlations between general happiness and life satisfaction

High correlation between general happiness and satisfaction with personal relationships;

high correlation between life satisfaction and satisfaction with finances.

Saris, Willem E., and Anna Andreenkova. 2001. “Following Changes in Living Conditions and Happiness in Post Communist Russia: The RUSSET Panel.” Journal of Happiness Studies 2: 95-109.

Contextual difference: homeownership Conventional definition of homeowner:

resident of “owner-occcupied household” Russia has highest rate of young

homeownership (ages 21-35) in Western & Eastern Europe according to this definition (about 85%).

But most are not living autonomously

“Homeownership” Rates Ages 21-35, with and without extended family

DETECTING MEASUREMENT NON-EQUIVALENCE

USING MULTIPLE INDICATORS AND

LATENT VARIABLE MODELS

Limitations of single measures Impossible to statistically test whether observed

difference (or similarity) is meaningful Example: consumer ethnocentrism

Russians are more likely than Canadians to agree with the statement: “There should be very little trading or purchasing of goods from other countries unless out of necessity.”

Possible sources of non-equivalence Translation Different interpretation, measuring different constructs

Saffu, Kojo, and John Hugh Walker. 2005. “An Assessment of the Consumer Ethnocentric Scale (CETSCALE) in an Advanced and Transitional Country: The Case of Canada and Russia.” International Journal of Management 22 (4): 556-571.

Latent variable approach

Consumer ethnocentris

m

should always

buy domesti

c

Foreign products only out

of necessity

Real (national

) buys domestic

Curbs should be put on all

imports

Foreigners should not be allowed

to sell

Buy domestic

Keep country working

Purchasing

foreign is un-

patriotic

(National) products, first, last, foremost

Foreign should

be heavily taxed

In latent variable model, oval represents latent construct, square represents manifest indicator.

Finding for Russia

Consumer ethnocentri

sm

should always

buy domesti

c

Foreign products only out

of necessity

Real (national

) buys domestic

Curbs should be put on all

imports

Foreigners should not be allowed

to sell

Buy domestic

Keep country working

Purchasing

foreign is un-

patriotic

(National) products, first, last, foremost

Foreign should

be heavily taxed

cultural

Economic

Example: National Identity Defined as beliefs about importance of potential

determinants of membership of the nation. Theory suggests 2 dimensions

Civic: residence, citizenship, respect for law/institutions

Ethnic/ascriptive: birthplace, descent, religion Common example in methodological literatureDavidov, Eldad. 2009. “Measurement Equivalence of Nationalism and Constructive

Patriotism in the ISSP: 34 Countries in a Comparative Perspective.” Political Analysis 17 (1) (December 21): 64–82. doi:10.1093/pan/mpn014.

Heath, Anthony, Jean Martin, and Thees Spreckelsen. 2009. “Cross-national Comparability of Survey Attitude Measures.” International Journal of Public Opinion Research 21 (3) (September 21): 293–315. doi:10.1093/ijpor/edp034.

Loner, Enzo, and Pierangelo Peri. 2009. “Ethnic Identification in the Former Soviet Union: Hypotheses and Analyses.” Europe-Asia Studies 61 (8): 1341–1370. doi:10.1080/09668130903134798.

Kunovich, Robert M. 2009. “The Sources and Consequences of National Identification.” American Sociological Review 74 (4) (August 1): 573–593. doi:10.1177/000312240907400404.

Sarrasin, Oriane, Eva G. T. Green, André Berchtold, and Eldad Davidov. 2012. “Measurement Equivalence Across Subnational Groups: An Analysis of the Conception of Nationhood in Switzerland.” International Journal of Public Opinion Research (October 18). doi:10.1093/ijpor/eds033.

Measurement model for pooled ISSP data

Source: Heath 2009

Civic

Ethnic

Born in nation

Nat’l ancestr

y

Nat’l religion

Life-long

resident

Citizen-ship

Respect laws

Speak languag

e

Feel nat’l

identity

Citizenship as ethnic

Source: Heath 2009

Countries where citizenship rules restrictive, ascriptiveCountries that are ethnically homogenousIncludes most former Soviet countries, Austria, Switzerland

Citizenship as civic

Source: Heath 2009

Countries where citizenship rules less restrictiveCountries with historical ethnic diversityIncludes Czech Republic, Spain, Australia, Israel

Formal tests for measurement invariance Configural invariance (weak): factor structure

equivalent (same items load on same latent variables) Metric invariance (strong): factor loadings equivalent:

necessary to compare relationships between constructs (e.g. regression coefficients).

Scale invariance (strict): factor loadings and intercepts equivalent; necessary to compare means

Partial invariance: at least two items load equally on each construct.Byrne, Barbara M., and Fons J. R. van de Vijver. 2010. “Testing for Measurement

and Structural Equivalence in Large-Scale Cross-Cultural Studies: Addressing the Issue of Nonequivalence.” International Journal of Testing 10 (2): 107–132. doi:10.1080/15305051003637306.

Cheung, Gordon W. 2008. “Testing Equivalence in the Structure, Means, and Variances of Higher-Order Constructs With Structural Equation Modeling.” Organizational Research Methods 11 (3) (July 1): 593–613. doi:10.1177/1094428106298973.

Steenkamp, Jan‐Benedict E. M., and Hans Baumgartner. 1998. “Assessing Measurement Invariance in Cross‐National Consumer Research.” Journal of Consumer Research 25 (1) (June 1): 78–107. doi:10.1086/209528.

What to do when invariance not achieved Delete problematic groups or countries

from comparison set Delete problematic items from

measurement model Settle for partial invariance: configural

invariance, plus at least 2 indicators per latent variable with equal loadings and/or intercepts

Problem with formal tests of invariance Formal strictness can undermine substance “Just as cross-national researchers

recognize that indicators may need non-literal translation to maximize the comparability between countries, constructs also may need non-literal translation.” (Medina et al 2009)

Achieving functional equivalence Statistical approaches

“Locally-conditioned models”: control for reasons for invariance (Medina 2009)

Introduce contextual predictor variables – i.e. directly model cross-group differences (Davidov 2012)

Qualitative context Investigate sources of invariance through

cognitive interviews, observation (Carnaghan 2011)Davidov, Eldad, et al. 2012. “Using a Multilevel Structural Equation

Modeling Approach to Explain Cross-Cultural Measurement Noninvariance.” Journal of Cross-Cultural Psychology 43 (4): 558–575. doi:10.1177/0022022112438397.

Carnaghan, Ellen. 2011. “The Difficulty of Measuring Support for Democracy in a Changing Society: Evidence from Russia.” Democratization 18 (3): 682–706. doi:10.1080/13510347.2011.563113.

RESPONSE STYLE BIAS

Response styles Tendencies in responses, independent of true

belief. Types of response styles

Tendency to agree (acquiescence bias) Tendency to moderate responses Tendency to extreme responses

Create bias in likert scales (e.g. agree-disagree), yes/no questions

Problematic when tendency to acquiesce varies across groups to be comparedHarzing, Anne-Wil. 2006. “Response Styles in Cross-national Survey Research A 26-

country Study.” International Journal of Cross Cultural Management 6 (2) (August 1): 243–266. doi:10.1177/1470595806066332.

Kieruj, Natalia D., and Guy Moors. 2013. “Response Style Behavior: Question Format Dependent or Personal Style?” Quality & Quantity 47 (1) (January 1): 193–211. doi:10.1007/s11135-011-9511-4.

Tobi, Hilde, and Jarl K. Kampen. 2013. “Survey Error in an International Context: An Empirical Assessment of Cross-cultural Differences Regarding Scale Effects.” Quality & Quantity 47 (1) (January 1): 553–559. doi:10.1007/s11135-011-9476-3.

Example: acquiescence in Kazakhstan Survey experiments on tendency to agree

among Kazakh vs. Russian respondents in Kazakhstan

Javeline, Debra. 1999. “Response Effects in Polite Cultures: A Test of Acquiescence in Kazakhstan.” The Public Opinion Quarterly 63 (1) (April 1): 1–28. doi:10.2307/2991267.

Possible causes of acquiescence bias Cultural significance of hospitality,

deference, avoiding offense Uncertainty about answer; assume

statements contain cues about correct answer

Cognitive burden Must infer counterarguments If agree with inferred counterargument, must

disagree with statement (yes/no) Satisficing due to fatigue, disinterest

Example: freedom versus order Version A: “People should be free to say whatever they

want, even if what they say increases tensions in society.” Version B: “Public order should be maintained above all,

even if it requires limiting freedom of speech.” Version C: “Certain people think that it is better to live in a

society with strict order, even if it requires limiting freedom of speech. Others think that people should be free to say whatever they want, even if what they say increases tensions in society. Which view is closer to your own? Do you feel this way strongly or only somewhat?”

Response options for versions A & B are: strongly agree/ somewhat agree/ somewhat disagree/ strongly disagree

Version C is a “forced choice” format question

Evidence for acquiescence bias

freedom -- very freedom -- somewhat

order -- somewhat

order -- very0

5

10

15

20

25

30

35

40

45

A: pro-freedomB: pro-orderC: forced choice

In absence of bias, proportions should be the same for all 3 versions.Acquiescence bias overstates support for freedom in version A, understates

support for freedom in version B.

Comparison of Kazakhs & Russians

freedom -- very

freedom -- somewhat

order -- somewhat

order -- very

0

5

10

15

20

25

30

35

40

45

freedom -- very

freedom -- somewhat

order -- somewhat

order -- very

0

5

10

15

20

25

30

35

40

45

KazakhRussian

VERSION A VERSION B

VERSION C

CONCLUSION:Acquiescence bias inVersion Aunderstates differencesBetween groups

freedom -- very

freedom -- somewhat

order -- somewhat

order -- very

0

5

10

15

20

25

30

35

40

KazakhRussian

What to do Avoid likert scales (esp agree/disagree) Use forced choice formats If using likert scale: Randomize direction of

question