![Page 1: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/1.jpg)
Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem
Comments
Jaki S. McCarthySenior Cognitive Research Methodologist
US Department of AgricultureNational Agricultural Statistics Service
WSS Seminar, December 2, 2014
![Page 2: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/2.jpg)
Another perspective on interviewer falsification
![Page 3: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/3.jpg)
Why do interviewers falsify?
• Understanding why can:
– Help identify relevant measures for models
– Help develop strategies to prevent falsification
![Page 4: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/4.jpg)
Falsifying to max $/min effort
• Most indicators have this underlying assumption (i.e. easy answers, shorter answers, rounding, etc.)
• Can we use data mining to identify other (less obvious) indicators?
![Page 5: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/5.jpg)
Falsifying to Meet Deadlines
• Do indicators change?• Maybe completion dates are important here
![Page 6: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/6.jpg)
Other reasons to falsify?
• Deliberate fabrication/data misrepresentation
• Fatigue
• Perceived reduction in respondent burden
• Would indicators be the same?
![Page 7: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/7.jpg)
Do these inform potential indicators/falsification model inputs?
• Speed indicators (length of interviews, completed interviews/day, etc.)
• Item nonresponse rates• Edit rates• Contact histories• GPS tracking
![Page 8: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/8.jpg)
How do we prevent falsification?How do we change motivation?
• Intrinsic versus extrinsic motivation• Employee (and respondent) engagement
![Page 9: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/9.jpg)
Extrinsic Motivation
• Should we pay interviewers more?• How much do you have to pay to ensure data
won’t be falsified?
• “Because of what they are paying me, I‘m going to collect the most accurate data possible.”
![Page 10: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/10.jpg)
Extrinsic Motivation
• Interviewers may falsify because they think no one is checking and it doesn’t matter
• Knowing that QA procedures are in place, and work is monitored can help here
• “Because I know they are checking my work, I’m going to collect the most accurate data possible.”
![Page 11: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/11.jpg)
Intrinsic Motivation
• How else can we motivate interviewers?
• “Because _____________________________, I’m going to collect the most accurate data possible.”
![Page 12: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/12.jpg)
How to get interviewers invested in the process
• Minimize Us versus Them– Supervisors/Monitors versus interviewers– HQ versus field– Data collectors versus data providers
• Value of the agency
• Value of the work
“Because __________________, I’m going to collect the most accurate data possible.”
![Page 13: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/13.jpg)
This extends to respondents too!
• Many of the indicators would flag poor quality data provided by respondents
• Why do respondents want to provide good quality data?• How can we improve the quality of respondents’
inputs?
• Will INTs who are good at gaining cooperation (i.e. getting cooperation from “hard to reach” units) look like they are collecting lower quality data?
![Page 14: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/14.jpg)
What can we do to get the right answer in that blank?
• Need to invest in – Training– Employee engagement– Communication up and down the chain
• Ultimate goal is to have only unintentional errors to detect
![Page 15: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/15.jpg)
Comments on Winker’s paper
![Page 16: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/16.jpg)
Curb-stoning as Fraud Detection ProblemAdvantages to this approach?
• Why not a classification problem?• Why not score interviewers using an index of
indicators?• How about scoring interviews and verifying
cases (not interviewers), or following up interviewers with highest percent of suspicious records?
• Is this an outlier detection problem?
![Page 17: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/17.jpg)
Objective way to narrow focus
• How to target scarce resources
• But as in other fraud detection problems, likely doesn’t go far enough (i.e. need to detect at much lower rates than 20% falsifiers with 70% falsification rate)
![Page 18: Curb-stoning, a Too Neglected and Very Embarrassing Survey Problem Comments Jaki S. McCarthy Senior Cognitive Research Methodologist US Department of Agriculture](https://reader036.vdocument.in/reader036/viewer/2022070412/56649ea45503460f94ba89c0/html5/thumbnails/18.jpg)
How can this method be extended?
• Are there other variables beyond indicators that might be useful in classifying falsifiers?– Data relevant indicators (time stamps, edit rates)– Person relevant indicators (INT characteristics –
Yes, I realize we are getting into dicey territory!)
– This would require “real” data – i.e. cannot be done with simulated data