The Total Survey Error Paradigm andChallenges to its Application in the Arab

World

Kathrin Thomas1 Michael Robbins1,2

1 Princeton University

2University of Michigan

5 June 2018

Where to begin...

"Honest discussions about the extent, sources of, ansolutions for quality issues [...] are needed. Regionallyspecific issues [...] raise concerns about social desirabilityand underscore the need for methodological research.Technological advances [...] o�er possibilities forreal-time monitoring and [...] research [...]. Yet apart froma handful of studies on interviewer e�ects, anchoringvignettes, and a few other topics, almost no researchsystematically assesses the impact of the surveymethods used on data quality in the Arab world."

(Benstead, 2018)

Puzzle

Is the Arab survey world just a few decades behind?Surveys and polling fairly new: beginning in 1980s,but real interest since the Arab uprising 2011(Benstead, 2018)While core principles of Total Survey Error (TSE) arefollowed, holistic concept not (yet) recognisedSurvey practice in the Arab world may also makeapplication of TSE more di�cult

The TSE

Construct

Measurement

Response

Edited Data

Population

Sampling Frame

Sample

Respondents

Survey Statistic

Validity

Measurement Error

Processing Error

Coverage Error

Sampling Error

Nonresponse Error

M E A S U R EM E N T R E P R E S E N T A T I O N

(Groves & Lyberg, 2010)

Representation

Potentially poor coverage due to past or ongoingsecurity treatsAbsence of transparent sampling frames and/oroutdated censusesLimited attention given to nonresponse andincomplete filesData fabrication, i.e, "Intentional departure from [...]sampling procedures by any member of the surveyproject" (Robbins, 2018)

Measurement

Translation: Arabic 6= Arabic 6= ArabicInterviewer e�ects on response or item nonresponse(e.g., Benstead and Maluche, 2014; Benstead,2014a, 2014b; Blaydes and Gillum, 2013; Koker,2009)Third party presence (Benstead, 2014)Data fabrication, i.e., intentionally deviating frominstructions; intentionally misrecording the answergiven by a respondent; or intentionally creating data(Robbins, 2018)

The TSE...again

Construct

Measurement

Response

Edited Data

Population

Sampling Frame

Sample

Respondents

Survey Statistic

Validity

Measurement Error

Processing Error

Coverage Error

Sampling Error

Nonresponse Error

New/different kinds of I & R Effects

Outdated Censuses;

mobile populations

Alternative Sampling Strategies

Absence of

Sampling Frames

M E A S U R E M E N T R E P R E S E N T A T I O N

Fabrication Fabrication

(Groves & Lyberg, 2010)

Data and Methods

Arab Barometer III and IVRepresentation: Contact attempts, data fabricationMeasurement: INR and response bias due tointerviewers and third party presence

Interviewer Gender

Example JordanInterviewer gender not included in the AB IV, butinterviewer names available...in ArabicThree coders (2 Arabic speakers, 1 non-Arabicspeaker) coded names into gender

κ varied from 0.37 to 0.70Non-Arabic speaker seemed to be least reliable; butalso the Arabic speakers had discrepanciesMajor challenges unisex names and context

Interviewer GenderAB IV

Interviewers in the Arab region predominantlyfemale (AB IV: 72 percent)

51

55

63

78

80

87

89

0 20 40 60 80 100Percent

Algeria

EgyptMoro

ccoTu

nisiaLe

bano

nJo

rdanPale

stine

Source: Arab Barometer Wave IV

Proportion of interviews conductedby female interviewers

Interviewer ReligiosityApprox. three quarters of the female interviewerswore a hijab during the survey interview (77.0percent)

Country Percent

Algeria N/AEgypt 100Jordan 96.3Lebanon 28.5Morocco N/APalestine 94.2Tunisia 29.1

Table: AB IV Proportion of Female Interviewers wearing aHijab

Third party presence

Almost every other interview was attended by athird person (44.6 percent)

12

24

48

55

58

59

62

0 20 40 60 80 100Percent

Leba

nonMoro

ccoEgy

ptAlge

riaJo

rdan

Tunis

iaPalestin

e

Source: Arab Barometer Wave IV

Proportion of interviews conductedwith third person presence

Representation: Coverage, Sampling Error, &Unit Nonresponse

Little information about any of these aspects of theTSE

Coverage problematic, esp. in less stable countriese.g., field report for Iraq AB III indicates that only"more than 66 percent of the population of Iraq" arerepresented

Sampling error problematic, esp. due to lack oftransparent informationNo collection/no provision of UNR filesResponse rate largely AAPOR standard, but not forall countries

e.g., field report for AB IV Palestine suggest 91percent response rate (response/response+refusal)

Representation: Contact Attempts

Example TunisiaUp to 8 contact attempts before UNR is loggedOn average, female interviewers need approximatelyone fewer contact attempt to interview comparedto male interviewersT-test reveals statistically significant di�erence incontact attempts across interviewer gender (t=2.7;p<0.01)

Representation: Suspected Data FabricationExample 1

Representation: Suspected Data FabricationExample 2

Representation: Suspected Data Fabrication

AB III AB IV

100 percent match 4 095 percent match 161 090 percent match 336 485 percent match 565 18

Total no. of suspicious cases 1,066 22

Total no. of observations in survey 14,809 9,000

Table: Suspected Data Fabrication AB III and AB IV

Representation: Implications

Keep more detailed records of sampling proceduresand framesRequest UNR files by contractRequest calculation of AAPOR RR by contractDevelopment of a live data checking tool that flagsup suspicious cases to prevent fabrication

Measurement: Interview Length

Example TunisiaShortest interview just over 30 minutes; longestinterview almost 2 hoursInterviews conducted by female interviewers tend tobe longer

T-test reveals statistically significant di�erence intotal interview length across interviewer gender(t=2.7; p<0.01)

Measurement: Item Nonresponse: Americansgood people? (1)

Overall, quite high INR (16.7 percent), substantivecountry variation

6

10

11

16

21

28

29

0 10 20 30 40 50Percent

Leba

nonPale

stine

Jorda

nAlge

riaTu

nisiaMoro

ccoEgy

pt

Source: Arab Barometer Wave IV

Proportion of INR

Measurement: Item Nonresponse: Americansgood people? (2)

Interviewer GenderMale interviewers appear to have higher INR(χ2=22.7; p<0.01) – all AB IV countries, but Algeria

Third Person PresenceThird person presence appears to increase INR(χ2=17.2; p<0.01) – esp. Algeria, Morocco, and Tunisia

Measurement: Interviewer Gender & Women’sRights (1)

While female interviewers seem to evoke moreagreement with the question whether women couldbecome prime minister/president (χ2=67.8, p<0.01)– esp. Jordan, Palestine, Tunisia.......male interviewers seem to evoke more agreementwhen asking if men should have the final say infamily decisions (χ2=61.6, p<0.01) – all AB IVcountries, except Tunisia

Measurement: Interviewer Religiosity & Women’sRights (1)

It seems to be more likely that respondents disagreewith the statement "women should be able tobecome prime minister/president" when the femaleinterviewer wears a headscarf (χ2 =23.4, p<0.01) –esp. Jordan, Palestine, TunisiaRespectively, wearing a hijab appears to evoke moreagreement with the statement that men shouldhave the final say in family decisions (χ2=8.5, p<0.01)– all AB IV countries, except Tunisia

Measurement: Third Person Presence &Women’s Rights (1)

Respondents seem to be more inclined to disagreewith the statement that women should be able tobecome prime minister/president when a thirdperson is present (χ2 =5.0, p<0.05) – esp. Algeria,Jordan, MoroccoBut, presence of a third person seems to lead tomore agreement with the statement that menshould have the final say in family decisions (χ2=6.3,p<0.01) – esp. Algeria, Morocco, Tunisia

Measurement: Implications

Design of a more extensive interviewerquestionnaire by the end of the survey interviewRequest for provision of core information aboutinterviewers (ID, name, gender, etc.) from the fieldagenciesInterviewer questionnaire capturing similarinformation (sealed envelope approach) atinterviewer trainingSwitch to Computer Assisted Personal Interviews in(almost) all countriesRequest for more detailed field reports detailingfieldwork procedures plus relevant records

Is the Arab survey world just a few decadesbehind?

Arab survey world is in transitionBenstead (2018) proposes an extensive agenda formethodological research on the Arab world, but arewe ready to walk the walk...yetThe AB has started collecting relevant data toempirically and systematically investigaterepresentation and measurement errorThe AB V will collect most of these data, so staytuned...

Contact Information

Kathrin Thomas:kathrint@princeton.edu

Michael Robbins:mdr7@princeton.edu

www.arabbarometer.org@arabbarometer@ArabBarometer