Cognitive Testing of Survey

Translations: Does Respondent

Language Proficiency Matter?

Patricia Goerman and Mikelyn Meyers, U.S. Census BureauHyunjoo Park and Mandy Sha, RTI International

Alisu Schoua Glusberg, RSS

Presented at the 70th annual conference of the American Association for Public Opinion Research (AAPOR)

Hollywood, FL: May 14-17, 2015

1

Overview of the Issue

Rule of thumb: survey translations should be tested with monolingual respondents

Assumptions:

Monolinguals are the intended users

Bilinguals maybe more likely to understand “bad” or overly literal translations:

For example: group home (hogar de grupo)

home schooling (enseñanza en el hogar)

2

Why Does it Matter?

More costly and time consuming to restrict testing to only monolingual respondents

Difficulties with recruiting/interviewing monolinguals

Distrust/lack of understanding of purpose

Correlation with lower income status

3

Review of the Literature

Survey Pretesting with monolinguals v. bilinguals -very little empirical research

(Park, et al. 2014)

Monolinguals: Difficulty with cognitive interview process (Park, et al. 2013; Pan et al. 2010; Goerman, 2006)

Differences between monolinguals and bilinguals in cognitive processes

(Bialystok, 2007; Hernandez and Bates 2001; Marian et al. 2009)

Monolingual status may be a correlate of other variables, e.g., educational attainment

(Ridolfo and Schoua Glusberg, 2011)

Important to include bilinguals in testing to account for differences in acculturation and other demographic characteristics

(Willis and Zahnd 2007)

4

Definitions

Cognitive testing: One-on-one interviews to evaluate whether respondents interpret, comprehend and respond to survey questions as intended.

Usability testing: One-on-one interview to study whether online questionnaire can be answered effectively, efficiently and with satisfaction by target respondents (Wang, 2015)

5

2014 Census Test Spanish Internet

Cognitive/Usability Testing

Origin of present study

Quick turnaround project: Test Spanish-language Census 2014 test instrument with 10 Spanish speakers

Recruitment goals Different national origins

Internet experience

Monolingual or Spanish-dominant, bilingual respondents to best test the translation

6

Preliminary Findings/ Impressions

Correlation between bilingual ability, education level and internet/computer experience

Accidental recruitment:

Non-computer literate, monolingual

Computer savvy, fluent English speakers

Issues of mode interacting with other respondent characteristics. How does language proficiency fit in with other considerations?

7

Design of Empirical Research

Decennial Census testing across modes and languages, large ongoing contract

Double the number of respondents so that half are monolingual, half bilingual

Spanish testing:

Paper, internet, interviewer administered form

Chinese, Korean, Vietnamese, Russian, Arabic Fillable forms: internet/paper, self administered mode

8

Research Questions for Overall

Project

Do monolingual and bilingual respondents help to “uncover” same number and types of issues?

Are there differences by mode?

Are there differences by language?

9

Early Results: Spanish, Interviewer-

Administered Instrument

Respondents asked about speaking, reading ability, dominant language in screening

39 Spanish-speaking respondents

19 monolingual

20 bilingual

Coding of interview summaries (3 coders)

Whether probe administered

Whether understood concept as intended

10

Respondent CharacteristicsDemographic Characteristics Monolinguals Bilinguals

GenderMale 9 8

Female 10 12

Age

18-34 6 9

35-44 3 6

45-54 8 3

44-64 1 2

65+ 1 0

Education*

Less than HS 10 0

HS grad/GED 7 16

College + 1 4

Years in US**

Less than 5 7 4

6-15 years 10 7

16+ 2 6

*Education level was missing for 1 monolingual respondent**Years in US was missing for 3 bilingual respondents

11

Results: Concepts Monolinguals

Misunderstood More Frequently

12

Concept

Monolinguals Bilinguals

% Misunderstood N % Misunderstood N

Foster child 94% 18 53% 19

Military assignment 60% 10 25% 12

Afroamericano 47% 15 19% 16

Own with mortgage 41% 17 15% 20

Live/stay somewhere else 21% 14 0% 15

Group home 87% 15 80% 10

Owner/renter name 40% 15 36% 11

% Misunderstood is calculated as : # who misunderstood a# who were asked probe (N)

Concepts Bilinguals

Misunderstood More Frequently

13

Concept

Monolinguals Bilinguals

% Misunderstood N % Misunderstood N

Housemate/roommate 43% 14 73% 15

Own free and clear 0% 18 20% 20

Seasonal home 50% 14 55% 11

Tenure question 16% 19 21% 19

Confidentiality 17% 6 21% 14

Indigena de las americas 50% 14 54% 13

Understanding of “Foster Child”

by English-Speaking Ability

10%

63%

90%

37%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Understood, N = 10* Misunderstood, N = 27

Speak English well /very wellSpeak English not well /not at all

14

*Data regarding understanding of this concept were unavailable for 2 respondents out of 39

NOTE: “Foster child” was translated as “Hijo(a) de crianza del programa Foster del gobierno”

Summary of Findings

Bilinguals caught most of the same problems as monolinguals but often less frequently

Depending on the size of a bilingual sample, researchers may or may not see an issue

One term was more problematic for bilinguals than monolinguals (Free and clear). Testing with only monolinguals may have masked this issue

Including only bilinguals in interviewer administered (CAPI) testing may work as a cost saving measure but some issues maybe lost

15

Next Steps

Calculate intercoder reliability and recode as necessary

More analysis of nuances in the types of problems identified

Analysis of additional modes in Spanish: Internet (data collected)

Paper questionnaires (pending funding)

Examination of monolingual v. bilingual respondents in 5 additional languages (pending funding)

Add analysis of English speakers for comparison

Look at “levels” of bilingualism

16

Cognitive Testing of Survey

Translations: Does Respondent

Language Proficiency Matter?

Patricia Goerman and Mikelyn Meyers, U.S. Census BureauHyunjoo Park and Mandy Sha, RTI International

Alisu Schoua Glusberg, RSS

For more information: E-mail: Patricia.L.Goerman@census.gov

Disclaimer: This presentation is intended to inform people about research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.

17