the different roles of interviewers: how does interviewer personality affect respondents’ survey...
TRANSCRIPT
The different roles of interviewers: How does interviewer personality
affect respondents’ survey participation and response behavior?
Michael Weinhardt (DIW Berlin)
International Total Survey Error WorkshopTällberg 2009
The German Socio-Economic Panel Study is a service unit of the
2
Multiple Error Sources
Measurement Error
Interviewer Characteristics:• Adherence to int. protocol• appropriate probing• accurate keying
Respondent Characteristics• Acquiescence• Extreme respsonse style
Item Characteristics:• Social Desirability• Privacy
Unit Nonresponse• No contact, no opportunity• Lack of time and/or motivation• Fear or maybe even shame to
participate
Different Interviewer characteristics interact with different error sources differently
One characteristic may reduce one type of error but increase another
→ conflicting priorities when recruiting interviewers
3
Survey Error and Interviewer Personality
Conscientiousnessneat, orderly and meticulous
ExtraversionSociability, enthusiasm, outgoing
character
AgreeablenessGentle, kind, seeking consultation,
not confrontation
Openness to ExperienceOpen for intellectual stimulation; new and unconventional ideas
NeuroticismEmotional instability, nervousness
Conscien-tiousness
Extraversion
Openness
Neuroticism
Int’s performance
Social Desira-bility Effects
Privacy Effects
Satisficing
+
+
-
-
-
-Agreeable-ness
Unit Nonresponse
4
Data 1: German Socio-Economic Panel Study
• “German PSID”: Nationally representative longitudinal panel study of private households and individuals
• Annual since 1984: 12,000 households; 22,000 persons
• Two-step sampling design: 1. Register sampling of communities (polling districts)2. Random route procedure
• Mixed Modes: – Personal Interviewing whenever possible (25% PAPI, 25% CAPI)– 36% self-administered (Interviewer present)– 14% postal
5
Data 2: GSOEP Interviewer Survey
• Mail survey of all interviewers of the 2006 SOEP wave
• Response rate: 94%; N=552 interviewers
• Update on interviewers’ demographics and other characteristics
• Systematic feedback on interviewers’ work experience: motivation, workload, opinion on incentives etc
• Self-rated measures of attitudes, values, beliefs etc.
• Same question format as in SOEP questionnaires regular for respondents
• Data linkage to individual level respondent data of the 2006 wave
• Measure of personality: Shortend Big Five Inventory (BFI)
– Also in SOEP 2005 → for both respondents and interviewers available
6
Interviewers’ Big Five
MeanSt.d.
52.71 7.90
50.89 7.72
53.31 8.21
54.16 7.46
44.57 9.16
7
Analysis: Model Specification
Two level hierarchical regression models, face-to-face interview cases only
Dependent variables: Measurement Error indicators
Independent variables: Big Five variables
Personality congruence between interviewer and respondent
Interviewer level covariates: Key demographics, experience as an interviewer, workload
Respondent level covariates: Key demographics (age, gender, region, education, whether self-employed),
experience with SOEP
8
Looking at Measurement Error: Results and Challenges
Findings:
• Interviewer conscientiousness reduces overall item nonresponse
• Personality congruence influential across a range of measures
• Interviewers’ personality dimensions are significantly related to (some) indicators of measurement error
But:
• Difficult to hypothesize links between personality dimensions and measurement error – lack in theory
• Single dimensions of interviewers’ personality show no consistent links to measurement error across a range of measures
• Unexpected directions of effects
9Significance: * = 10% level; ** = 5% level ; *** 1% level
Interviewer performance and overall data quality Logistic Regression Negative Binominal Regression
Interviewer Level Covariates
Overall Item NR (binary) Overall Item NR (Count)
Odds Ratio Std. Err. Coefficient Std. Err.
Experience as Interviewer 1.002 0.010 0.000 0.003
Gender 0.767** 0.103 -0.249*** 0.042
Region 0.791 0.257 -0.149 0.119
Age 1.015** 0.007 0.013*** 0.002
Workload 0.997 0.002 -0.002*** 0.001
Openness 1.006 0.008 -0.001*** 0.001
Conscientiousness 0.984** 0.008 0.010*** 0.002
Extraversion 1.012 0.008 -0.015 0.003
Agreeableness 0.997 0.009 0.002*** 0.003
Neuroticism 1.003 0.007 -0.007 0.003
Pers. Congruence 0.990*** 0.003 -0.001*** 0.002
# of Items 0.000 0.000
Rho 0.238 0.021
Number of Obs 8975 8975
Number of Int's 417 417
10Significance: * = 10% level; ** = 5% level ; ***= 1% level
Privacy effects: NR income items Logistic Regression
Interviewer Level Covariates
Gross Income NR (binary)
Net Income NR (Count)
Salary NR (Count)
Odds Ratio Std. Err. Odds Ratio Std. Err. Odds Ratio Std. Err.
Experience as Interviewer 1.005 0.023 0.988 0.020 1.004 0.020
Gender 0.966 0.280 0.871 0.236 1.172 0.302
Region 2.823 1.803 0.941 0.576 1.454 0.909
Age 0.981 0.014 0.978 0.014 0.985 0.013
Workload 0.995 0.005 0.997 0.004 0.991* 0.005
Openness 1.013 0.018 0.986 0.016 0.999 0.016
Conscientiousness 0.984 0.018 0.985 0.017 0.987 0.017
Extraversion 1.009 0.017 1.023 0.017 1.030* 0.016
Agreeableness 1.015 0.020 1.019 0.019 1.014 0.018
Neuroticism 1.028* 0.015 1.019 0.014 1.009 0.013
Pers. Congruence 0.992 0.007 0.980** 0.008 0.982** 0.009
Rho 0.507 0.146 0.388 0.142 0.333 0.049
Number of Obs 4540 4604 4224
Number of Int's 369 372 366
11Significance: * = 10% level; ** = 5% level ; ***= 1% level
Social Desirability: Alcohol, cigarettes and weight
Logistic Regression OLS Regression Logistic Regression
Interviewer Level Covariates
Regular alcohol consumption (binary)
Cigarettes smoked per day (continuous)
Round values weight variable (binary)
Odds Ratio Std. Err. Coefficient Std. Err. Odds Ratio Std. Err.
Experience as Interviewer 1.001 0.009 0.017 0.017 1.001 0.005
Gender 0.897 0.099 -1.032** -1.032 1.066 0.064
Region 1.113 0.340 -0.711 -0.711 0.895 0.158
Age 1.012** 0.006 -0.011 -0.011 0.998 0.003
Workload 1.001 0.002 -0.005 -0.005 0.999 0.001
Openness 0.987** 0.006 0.047* 0.047 1.011*** 0.004
Conscientiousness 0.989 0.007 0.028 0.028 1.006 0.004
Extraversion 0.993 0.006 0.024 0.024 1.005 0.004
Agreeableness 0.992 0.007 -0.049* -0.049 1.003 0.004
Neuroticism 1.001 0.006 0.009 0.009 0.996 0.003
Pers. Congruence 0.998 0.003 -0.030 -0.030 0.991*** 0.002
Rho 0.109 0.016 0.013 0.018 0.005
Number of Obs 8927 2463 8907
Number of Int's 416 342 417
12Significance: * = 10% level; ** = 5% level ; ***= 1% level
Acquiescence and Extreme Response style
OLS Regression Negative Binomial Regression
Interviewer Level CovariatesIndicator Acquiescence Indicator Extreme Response
Coefficient Std. Err. Coefficient Std. Err.
Experience as Interviewer -0.010 0.010 0.001 0.003
Gender -0.166 0.136 0.038 0.041
Region 0.437 0.351 0.026 0.097
Age 0.013** 0.007 -0.004** 0.002
Workload 0.003 0.002 -0.002*** 0.001
Openness 0.015* 0.008 -0.004* 0.002
Conscientiousness -0.005 0.009 0.010*** 0.003
Extraversion 0.003 0.008 0.000 0.002
Agreeableness 0.006 0.009 0.000 0.003
Neuroticism 0.019*** 0.007 -0.011*** 0.002
Pers. Congruence -0.032*** 0.003 0.004*** 0.001
Rho 0.107
Number of Obs 8927 8975
Number of Int's 416 417
13
Attrition and unit Non-responsePreliminary analysis: same model set-up as before
• Linked 2006 data on interviewers and respondents to 2007 outcomes - binary variable: attriter or not
• No significant results: neither for big five nor for personality congruence
• However: overall item missingness is significantly related to attrition
Further thoughts on attrition analysis:
• What kind of effect would we expect?
• Original distribution of Big 5 among interviewers not known – self selection of interviewers possible
• Attrition due to personality effects may have happened before 2006 already
15
Similarity in Personality• Literature: similarity in personality is linked to affection and friendship, e.g.
personality congruence in married couples (Rammsttedt and Schupp 2009)
Personality congruence might also play an important role in the interviewer-respondent relationship
• Measurement: Euclidean distance between interviewers‘ and respondents‘ big five scores
• The sqareroot of the sum of squared differences in the big five factor scores
• Mean 50 ; Standard deviation: 10
• Premultiplied by -1 for ease of interpretation
2
1
( , ) ( )n
i ii
d X Y X Y