survey experiments
DESCRIPTION
Survey Experiments. Defined. Uses a survey question as its measurement device Manipulates the content, order, format, or other characteristics of the survey as a treatment. Methodological Issues. Missing Data Matching Both can be an issue in experiments other than surveys. Missing Data. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/1.jpg)
Survey Experiments
![Page 2: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/2.jpg)
Defined
• Uses a survey question as its measurement device
• Manipulates the content, order, format, or other characteristics of the survey as a treatment
![Page 3: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/3.jpg)
Methodological Issues
• Missing Data
• Matching
• Both can be an issue in experiments other than surveys
![Page 4: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/4.jpg)
Missing Data
• Some observations missing data on the DV or IVs
• If missing at random, not a problem to drop from the analysis
• But usually not missing at random• Deleting non-random missing causes bias
![Page 5: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/5.jpg)
Missing Data II
• Data can also be missing intentionally:• Some cases not “treated”• Possible to “guess” what would have
happened to a subject had they been in another treatment group– Allows within-subject comparison of two
treatments, the one they received and the one they could have received
![Page 6: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/6.jpg)
Solution: Imputation
• Suppose Yi = a + b1Xi1 + b2Xi2 +ei
• But Yi missing for some observations
• Xi1 and Xi2 not missing
• Regress Y on Xi1 and Xi2 for all non-missing observations
• Use b1 and b2 to calculate predicted Ypi
![Page 7: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/7.jpg)
Better Yet: Multiple Imputation
• Ypi is a predicted value with uncertainty
• Multiple imputation predicts multiple values for Yp
i drawn from a distribution of predicted values
• 5 or so predicted Ypi sufficient for inference,
no need for many• Gary King’s Amelia program available free on-
line
![Page 8: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/8.jpg)
Matching
• Experiments can be pre-matched to avoid large random sample
• Match subjects on important characteristics such as– Sex– Race– Age– Education levels– Other traits?
![Page 9: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/9.jpg)
![Page 10: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/10.jpg)
Matching
• Often necessary in field experiment when randomization more difficult to control
• propensity score is the probability of an observation being assigned to a particular treatment in a study given a set of known variables.
• Propensity scores reduce selection bias by equating groups based on these variables
![Page 11: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/11.jpg)
A Theory of Nonseparable Preferences in Survey Responses
![Page 12: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/12.jpg)
Question
• Why do people change their answers to survey questions if the order of questions changes?
• Does changing survey responses indicate that people do not have well-formed opinions
![Page 13: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/13.jpg)
![Page 14: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/14.jpg)
Theory
• Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue
• Separable Preferences: What a person wants on every issue is independent of what they get on other issues
![Page 15: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/15.jpg)
Measuring Nonseparable Preferences
![Page 16: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/16.jpg)
![Page 17: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/17.jpg)
![Page 18: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/18.jpg)
![Page 19: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/19.jpg)
Method
• Randomize the order of pairs of survey questions– For some issues, aggregate responses different
across question order
• Each subject answers questions in order– Issue 1 then Issue 2– Issue 2 then Issue 1
![Page 20: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/20.jpg)
Method
• Impute what subject would have answered had they heard questions in different order
• For each question we then haveYi (if first) – Yi (if second)
• One of these will be imputed for each person since they cannot answer a question both first and second in the order
• First study to analyze individual differences in question orders, not simply aggregate differences
![Page 21: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/21.jpg)
![Page 22: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/22.jpg)
Conclusions
• Nonseparable preferences explain question order effects
• Political information level does not • Response instability not due to uninformed
respondents
![Page 23: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/23.jpg)
Are Survey Experiments Externally Valid?
JASON BARABAS and JENNIFER JERITAmerican Political Science Review
2010
![Page 24: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/24.jpg)
Question
• Many survey experiments expose subjects to different information to show effect of on responses
• In a survey experiment, subjects are a “captive audience” that must pay attention
• Do the same information effects appear in the real world
• Compare survey experiments with natural experiments
![Page 25: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/25.jpg)
Method
• Survey experiments give people to political information about immigration and medical care
• Pre-post survey also in field during change in medical insurance and immigration– Ask respondents which media sources they use
• Is the effect of information in the survey experiment as large as in the natural experiment?
![Page 26: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/26.jpg)
![Page 27: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/27.jpg)
![Page 28: Survey Experiments](https://reader033.vdocument.in/reader033/viewer/2022052603/56812dc1550346895d93014d/html5/thumbnails/28.jpg)