![Page 1: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/1.jpg)
Diana C. MutzUniversity of Pennsylvania
![Page 2: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/2.jpg)
Random sampling + Random assignment =
Population-Based Experiments
![Page 3: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/3.jpg)
Goal to increase methodological innovation and to expand the diversity of methods used across the social sciences
Particularly studies that benefit from combining the internal validity of experiments with the external validity of representative population samples.
![Page 4: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/4.jpg)
NSF-supported infrastructure project that provides social scientists with opportunities for innovative data collection
TESS uses two large-scale data collection instruments
Internet surveys Telephone Interviews
Our goal is to increase methodological innovation and to expand the diversity of methods used across the social sciences.
![Page 5: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/5.jpg)
Experiments as the Gold Standard for establishing causality:
“There are no complex statistical techniques that provide the power and elegance of a simple experimental design."
![Page 6: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/6.jpg)
“College sophomores may not be people.” (C. Hovland).
“A greater effort must be made to conduct research on persons from life stages other than late adolescence.”
“… I suspect that even middle-aged people, separated from family and friends and confronted with test-like materials on novel and artificial topics in an academic laboratory, would often behave like college students do.” (D. Sears).
![Page 7: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/7.jpg)
Allowed extensive pretesting. Investigators can listen in on
pretest interviews, and revise accordingly.
Respondents are sent pre-survey incentives and are paid for their interviews.
Timing can be coordinated with Internet data collection.
![Page 8: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/8.jpg)
Address-based sampling (to include cell-phone-only households) for recruitment.
Free notebook computer and internet access if needed so both internet and non-internet households are included
Payment for study participation Special subpopulations known in advance Multimedia: Pictures/Video/Audio Rapid Turnaround
![Page 9: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/9.jpg)
Decreasing investigator demand for this mode of data collection.
Limitations on experimental stimuli More difficult to recontact the same
respondents (greater attrition) Evidence on reliability of measures
generally better via internet Ability to randomly sample
specialized populations
![Page 10: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/10.jpg)
Any faculty member or graduate student in any social science department anywhere in the world.
Proposals may come from any substantive area in the social sciences as long as they utilize experimental or quasi-experimental designs.
Repeated use is encouraged.Researchers to build on previous findings in subsequent proposals.
![Page 11: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/11.jpg)
Any experiment or quasi-experimental study designed to explore substantive or methodological hypotheses within the social sciences.
Limits of request for sample and number of questions
Must make a clear contribution to knowledge. Particularly studies that would benefit from
combining the internal validity of experiments with the external validity of representative population samples.
![Page 12: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/12.jpg)
Proposals are limited to 5 pages in length. Minimal investment of time in proposal
preparation Encourages fast turnaround from reviewers
Proposals are submitted through an easy online process. With just a few clicks of the mouse, your
proposal enters the review process.
![Page 13: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/13.jpg)
Diverse Associate PIs oversee the review process…
Diversity of topics reduces potential contamination of respondents.Investigators need not share a survey interview.
![Page 14: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/14.jpg)
Capitalizes on economies of scale to provide more data to more investigators than would otherwise be possible for the same cost.
Encourages social scientists to use methodologies they would not otherwise use.
Encourages quicker pace of research progress
Promotes more policy-relevant research
![Page 15: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/15.jpg)
1. Experimental designs on policy-relevant populations
![Page 16: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/16.jpg)
Experimentally investigated what preventative care benefits were most important to consumers
Respondents: 500 with a minor
present in their home 500 without a minor
present in their home
Respondents choose among three health plans, with the possibility of opting into special vaccination coverage.
![Page 17: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/17.jpg)
Respondents randomly assigned to receive threatening
or non-threatening feedback on a general knowledge
test or a “masculine knowledge test:”
When it is sold in the US, what country does the beer Heineken come from?
What are the true measures of a “two by four”?
Who had back-to-back MVP seasons in the NFL besides Joe Montana?
Which filter on a car typically requires changing every 3000 miles?
What is considered a good maximum benching weight?
Which tool clamps onto an object and will stay clamped without being held in place?
![Page 18: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/18.jpg)
The conventional wisdom about infidelity based on student samples:
Men value sexual fidelity, whereas women value emotional fidelity.
The finding when research is based on a representative sample of adults:
Both men and women show more anger and blame over sexual infidelity, and both men and women show greater hurt feelings over emotional infidelity.
![Page 19: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/19.jpg)
Do OCD screening batteries work less well for African-Americans?
If so, why?
2 by 2 experiment with representative samples of European-Americans (50%) and African-Americans (50%), asked about ethnic and racial identity either before the OCD battery (to increase salience of race), or after the scales were asked.
![Page 20: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/20.jpg)
1) Same racial difference in average scores in representative samples as in college student sample. 2) Increasing the salience of race increases African-Americans’ OCD screening scores.
![Page 21: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/21.jpg)
1. Experimental designs on policy-relevant populations
2. Using advance knowledge of respondent characteristics
![Page 22: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/22.jpg)
1. Experimental designs on policy-relevant populations
2. Using advance knowledge of respondent characteristics
Representative sample of participants drawn from the population of individuals whose family income is less than $30,000/year
Representative sample of people residing in coastal areas
![Page 23: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/23.jpg)
![Page 24: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/24.jpg)
1. Experimental designs on policy-relevant populations
2. Using advance knowledge of respondent characteristics
3. More complex, subtle and engaging interactions with respondents
![Page 25: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/25.jpg)
A person is “laid off” because his or her company had to reduce its staff.
“Think for a moment about the person and then tell me how much government help, if any, that person should receive while looking for a new job.”
One vignette, 96 variations:The person is a [white/black] [male/female] in [his/her] [early 20’s/mid 30’s/early 40’s]. [He/She] is [single/a single parent/married/married and has children] and [is/is not] a dependable worker.
![Page 26: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/26.jpg)
![Page 27: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/27.jpg)
Random samples of Black and White men and women evaluate Black and White, male and female models.
Asked to indicate when model becomes “overweight,” and when “obese”
![Page 28: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/28.jpg)
![Page 29: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/29.jpg)
![Page 30: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/30.jpg)
![Page 31: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/31.jpg)
![Page 32: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/32.jpg)
![Page 33: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/33.jpg)
![Page 34: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/34.jpg)
![Page 35: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/35.jpg)
![Page 36: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/36.jpg)
![Page 37: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/37.jpg)
![Page 38: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/38.jpg)
Results:Female image perceived as overweight
and obese at significantly smaller body sizes than the male image.
Race of image did not matter, but race of perceiver did; White men judged bodies as obese and overweight at smaller body sizes than Black men
![Page 39: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/39.jpg)
Does seeing the aftermath of natural disasters affect willingness to insure against them?
Tornadoes, Hurricanes, Fires
![Page 40: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/40.jpg)
![Page 41: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/41.jpg)
Disentangling direction of causality:
Perceived quality of neighborhood
Level of social organization, interaction
![Page 42: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/42.jpg)
![Page 43: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/43.jpg)
Do the deaths of private security personnel (i.e., “contract soldiers”) affect Americans less than deaths of military soldiers?
Do they elicit less of a response?Do they help insulate support for the war
from the death toll?
![Page 44: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/44.jpg)
![Page 45: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/45.jpg)
1. Experimental designs on policy-relevant populations
2. Using advance knowledge of respondent characteristics
3. More complex, subtle and engaging interactions with respondents
4. Repeated contact over time
![Page 46: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/46.jpg)
Did Sotomayor’s confirmation change Hispanic attitudes toward government?Toward the Supreme Court?
Three groups interviewed after nomination, but before confirmation, then again after confirmation
![Page 47: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/47.jpg)
Nominated, but not confirmed Post-confirmation
![Page 48: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/48.jpg)
Nominated, but not confirmed Post-confirmation
![Page 49: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/49.jpg)
1. Experimental designs on policy-relevant populations
2. Using advance knowledge of respondent characteristics
3. More complex, subtle and engaging interactions with respondents
4. Repeated contact over time5. “Firehouse” studies in response to
unfolding events
![Page 50: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/50.jpg)
Respondents:
Georgia Residents
Half of Respondents, African-American
Half of Respondents, White
![Page 51: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/51.jpg)
1. Experimental designs on policy-relevant populations
2. Using advance knowledge of respondent characteristics
3. More complex, subtle and engaging interactions with respondents
4. Repeated contact over time5. Firehouse studies6. Improving measurement, especially
sensitive issues and behavioral outcomes
![Page 52: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/52.jpg)
Why such large discrepancies between the self-reports of heterosexual men and women?
Social desirability? Memory strategies?
Men - top of head multiplication Women – enumeration of individuals
![Page 53: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/53.jpg)
![Page 54: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/54.jpg)
![Page 55: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/55.jpg)
![Page 56: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/56.jpg)
Ball tossing game to induce feelings of rejection and social ostracism.
![Page 57: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/57.jpg)
Source of ideas and examples for population-based experimental designs
![Page 58: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/58.jpg)
![Page 59: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/59.jpg)
![Page 60: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/60.jpg)
Six thousand respondents answer two questions each.
Here is a list of things some people have done, please tell us how many you have done:
Written a letter to an editor of a newspaper Called into a talk radio program Visited a country outside of the United
States
![Page 61: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/61.jpg)
Bodenhousen, “The role of gender in mental illness stigma”
Corrigan, “Shame, Blame and Contamination: Mental Illness Stigma and the Family”
Use of between subjects design, large samples, and anonymous internet surveys help acquire sensitive data
![Page 62: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/62.jpg)
2 (masculine vs. general knowledge test) x 2(threatening vs. non-threatening feedback) experimental design.
Rs complete a 10-question test. They are randomly assigned to the masculine-knowledge test or the general-knowledge test.
After the test, Rs receive feedback on how they did. Rs are randomly assigned to receive threatening feedback or non-threatening feedback.
Dependent measures involve a series of questions about attitudes toward gay men.
![Page 63: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/63.jpg)
Masculine Knowledge TestWhen it is sold in the US, what country does the
beer Heineken come from? What are the true measures of a “two by four”?Who had back-to-back MVP seasons in the NFL
besides Joe Montana?Which filter on a car typically requires changing
every 3000 miles?What is considered a good maximum benching
weight?Which tool clamps onto an object and will stay
clamped without being held in place?
![Page 64: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/64.jpg)
Proposers may designate the field within which they would like their proposal reviewed.
After reading the proposal, a TESS Associate PI selects two reviewers appropriate to the topic and methods.
The reviewers are sent the anonymous proposal via email and respond within two weeks.
![Page 65: An Introduction to Population-based Survey Experiments](https://reader037.vdocument.in/reader037/viewer/2022102819/56814924550346895db65def/html5/thumbnails/65.jpg)
Continuous panel recruitment
Rotating panel with tenure of approximately 3
years Maximum of 1 survey per week; usually 3 surveys
per month involving a broad spectrum of topics and customers
Average 10-15 survey minutes in length per week, with some surveys of 25-35 minutes with incentives
Sample selection controls for prior survey taking to minimize panel conditioning