3/29/2016

1

Pros and Cons of

Non-Probability Sampling

Linda Owens

General information

� Please hold questions until the end of the presentation

� Slides available at www.srl.uic.edu/SEMINARS/Spring16Seminars.htm

� Please raise your hand so that I can see that you can hear me

2

3/29/2016

2

Focus of the webinar

�Overview of Probability Samples

�Definition of Non-probability Samples

�Examples of Nonprobability Samples

�Conclusions

3

Characteristics of Probability Samples

�A frame of all units in the target population can be constructed�Every unit in the frame has a nonzero probability of being selected�The probability of selection can be calculated for each sampled unit�Multistage samples or samples drawn with unequal probabilities can still be probability samples

�Necessary for making population inferences (a set of procedures that produces estimates about the characteristics of a target population and provides some measure of the reliability of those estimates—AAPOR task force).

4

3/29/2016

3

Short History of Probability Sampling

�Early approaches to surveys used censuses and quota samples

� Jerzy Neyman, 1934 paper in the Journal of the Royal Statistical Society

�First large scale probability sample in US—1939 Works Project Administration

�Acceptance by broad scientific community a result of two events:� 1936 Roosevelt-Landon presidential election

• Two small surveys predicted outcome correctly, while massive poll of 10 million did not

• Quantity ≠ Accuracy

� 1948 Truman-Dewey presidential election• Polls using quota sampling mistakenly predicted Dewey victory

• Probability sampling became the norm

5

Challenges with Using Probability Samples

�CoverageDecline of landline phones and increase in cell phones

�NonresponseResponse rates have been declining steadily over past 40 years

�CostDeclines in coverage and cooperation have increased costs

6

3/29/2016

4

Non-Probability Samples

� Increase in use in part because of:� increasing reliance on less expensive Web surveys� increasing amount of research on hard-to-reach populations

�No single framework encompasses all forms of non-probability sampling

� Includes a wide range of methods, from simple and straightforward, to complex

�Nomenclature confusing�Examples in this webinar are not exhaustive

7

Categories of Non-Probability Samples

� Convenience Samples

…a form of non-probability sampling in which ease of locating and recruiting participants

is the primary consideration

�Sample Matching

Match the sample to the control group or larger population on a number of characteristics

�Network Sampling

Link tracing--using social network connections to facilitate sampling

8

3/29/2016

5

Convenience Samples

� Intercept surveys (shoppers at mall, students on quad)

�Panels of volunteers (Opt-in panels, advertising for subjects)

�River samples (recruitment from websites)

�Observational studies (case control, cohort)

�Snowball sampling (as typically implemented)

9

Convenience Samples: Pros

�Cheap and easy to conduct

�Can be used in some circumstances� Pilot studies

� Exploratory studies

10

3/29/2016

6

Convenience Samples: Cons

�Statistical inference not possible because…� No way to calculate probability of inclusion

� Sample likely to be small and unrepresentative

� Often consist of volunteers, who may not be representative of target population

� Participation rates are low

�No theoretical basis or explicit set of assumptions for making estimates and for judging the accuracy of those estimates

11

Sample Matching

�Quota samples (Health of Nation Survey, many others)

�Random assignment

� Randomized controlled trials (intervention(s) compared to no intervention)

� Medical studies and clinical trials (comparison of interventions)

� Drugs, devices, treatment locations, weight-loss programs

�Quasi-experimental design (assess causation without randomization)

�Sample matching for surveys (YouGov)

12

3/29/2016

7

Sample Matching: Pros

�Can reduce bias (relative to non-matched non-probability sample) if: � characteristics (covariates) used for matching are related to outcome of interest

� sample and population have similar distribution on matching covariates

�Can provide good internal validity

�Randomized controlled trial is the gold standard in medical research

13

Sample Matching: Cons

�Problems with covariates� Including all relevant ones

� Balancing

�External validity--much emphasis on power and sample size, but little on generalizability

�“…no standard sample matching practices for collecting non-random, non-experimental data that generally support inferences to a larger population (AAPOR task force).”

14

3/29/2016

8

Network Samples

�Snowball sampling (as originally developed)—random sample, ask for referrals for others with same characteristics

�Multiplicity sampling (is a probability sample)—start with random sample and ask about others in respondent’s network

�Respondent-driven sampling—similar to snowball, but:� Dual incentives (completion and recruitment)

� Recruitment coupons

15

Network Samples: Pros

�Useful for drawing samples from hard-to-reach populations

�May be only way to target some populations

�Can draw inferences under tight set of assumptions

16

3/29/2016

9

Network Samples: Cons

�Only works when characteristic(s) of interest create contact with

others

�Valid inference requires a number of untestable assumptions

�Dependent samples → higher variance, lower effec�ve sample size

�Resulting sample influenced by initial sample

�Due to nature of target population, initial sample often not random

17

General Issues with Non-Probability Samples

�Weighting� Base weights can’t be calculated� Post-stratification weights cannot fix bad sample

�Statistical inference generally not possible�No equivalent to Total Survey Error�Issues of nonresponse and poor coverage that affect probability samples are also present in non-probability samples

�Less accurate than probability samples (Yeager, et al 2011)

18

3/29/2016

10

Survey AccuracyLow High

Non-probability samples

Probability samples

Some can be of higher quality than poorly done probability

samples

19

AAPOR Task Force: Some Conclusions & Recommendations

�No framework encompasses all non-probability samples

�Approaches fall on a continuum of quality

�Transparency in methodological reporting is essential

�Survey researchers need to develop a framework for evaluating quality

20

3/29/2016

11

AAPOR Task Force: Some Conclusions &

Recommendations (con’t)

�Inferences require modeling assumptions Sample sizes (by sampling frame if more than one was used) and a discussion of the precision of the findings. For probability samples, the estimates of sampling error will be reported, and the discussion will state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to weighting, clustering, or other factors. Disclosure requirements for non-probability samples are different because the precision of estimates from such samples is a model-based measure (rather than the average deviation from the population value over all possible samples). Reports of non-probability samples will only provide measures of precision if they are accompanied by a detailed description of how the underlying model was specified, its assumptions validated and the measure(s) calculated. To avoid confusion, it is best to avoid using the term “margin of error” or “margin of sampling error” in conjunction with non-probability samples (AAPOR Code of Professional Ethics & Practices--Standards for Disclosure).

�Modeling requires statistical expertise

21

Questions?

� Type questions in chat box.

22

3/29/2016

12

Thank You!

www.srl.uic.edu

23