Questions to Ask Before You Believe a Survey

Who Carried Out The Survey?

• Just because a survey on gun-control was conducted by the NRA doesn’t mean they were biased, but they certainly have reason to be

• Surveys are easier to believe when conducted by a neutral party that’s experienced in conducting samples

What was the Population?

• Whose opinions were being sought?• If you know the population then you know who

the survey results apply to and who they don’t.• A survey of Hawaiians won’t tell us much about

Indiana’s opinions.

How was the Sample Selected?• Voluntary Sampling• Convenience Sampling• Probability Sampling– SRS: Simple Random Sample– Stratified

How was the Sample Selected?Voluntary, Convenience or Probability

• Which has the least bias?• Using a Voluntary Sample, which type of

people are you most likely to get? Least?– A) Strong Positive Opinions– B) Apathetic People– C) Strong Negative Opinions

• What is the difference between SRS & Stratified?

Probability

Most-OverrepresentedLeast-Underrepresented

SRS – Take random people from the populationStratified – Break population into groups and then take an SRS from each group

How Large was the Sample?• More people = lower variability• If n < 10% of the total population (which it

almost always is) then…

– Population size doesn’t matter if n < 10% of Pop.• How many people do you need to survey to

have an error of 1%?

What was the Response Rate?• Low Response Rate = High Bias

How were the Subjects Contacted?

• An internet survey of where people get their news is likely to have a different response than a phone interview.

• Face-to-face interviews are likely to have a higher response rate than phone interviews

• Are there significant portions of the population that suffer from undercoverage because of the survey method?

When was the Survey Conducted?• Opinions change with time. During the

primary, for awhile the Republicans had a new front runner every month.

• Events influence opinions. Right after 9/11 most Americans would say Muslim Extremists are the greatest threat to their safety while domestic terrorism was the greatest threat right after the Oklahoma City Bombing.

What were the Exact Questions?

• Biased? – Should we restrict firearm sales to reduce crime?– Should we allow law-abiding citizens to own a gun

to protect their families?• Roanoke College did a survey of Virginians:– 80% believe in life after death– 87% believe in Heaven– Explain how 7% believe there is a Heaven, but no

life after death?