item nonresponse in a mail survey of young adults

Item Nonresponse in a Mail Survey of Young Adults

Luciano Viera, Jr., Scott Turner, and Sean MarshFors Marsh Group LLC

Overview Background

– RDD Telephone Surveys: Coverage Implications

Youth Poll Mail Study– Purpose and Methodology– Mail YP Frame Coverage – Data Collection Efficiency Comparison– Profile of Responders– Survey Estimate Comparison

Item Nonresponse

Incentive Experiment

Methodology Transition

2011 ITSEW – Québec, Canada

2

Background The US Department of Defense (DoD) conducts the Youth Poll (YP)

to track military propensity– Tracked over the past 35 years – Critical to maintain these trend lines– Random-digit-dial (RDD) survey design– Historically, estimates have been shown to be reliable and valid

For decades, RDD telephone surveying has been a cost-efficient way to survey the general public

– RDD surveys are typically interviewer-administered

Emerging trends are impacting the future viability of this methodology

– Decreasing coverage of telephone surveys– General decrease in survey research response rates– Reduced efficiency due to coverage and nonresponse issues


3

RDD Telephone Surveys: Coverage Implications

CDC study found 46% of 18- to 24-year-olds live in wireless-only households

Households with a landline are different from those without a landline

– Wireless-only households are more likely to be located in urban, metropolitan areas


4

2.9% 3.5% 4.4% 5.4%6.7% 7.7%

9.6%11.8%

12.6%14.5%

16.1%18.4%

21.1%22.9%

24.9%27.8%

1.6% 1.7% 1.5% 1.8%1.6%

1.7% 1.8% 1.7% 1.6% 1.9% 2.1% 1.7% 1.5% 1.7% 1.7% 1.8%2.8% 3.2% 3.7% 4.9% 5.8% 7.6%

8.6%11.6% 11.9%

14.4%

17.0%18.7%

21.3%

25.9%

29.0%31.8%

1.8% 2.0% 1.8% 2.3%1.5%

1.8% 1.9% 2.3% 1.7% 2.1% 2.5% 2.4% 1.7% 1.9% 1.7% 2.0%

0%

5%

10%

15%

20%

25%

30%

35%

Jan-Jun2003

Jul-Dec2003

Jan-Jun2004

Jul-Dec2004

Jan-Jun2005

Jul-Dec2005

Jan-Jun2006

Jul-Dec2006

Jan-Jun2007

Jul-Dec2007

Jan-Jun2008

Jul-Dec2008

Jan-Jun2009

Jul-Dec2009

Jan-Jun2010

Jul-Dec2010

Children with wireless service only

Children with no telephone service

Adults with no telephone service

Percentage of Adults and Children with Only Wireless or No Telephone Service

Adults with wireless service only

Source: Blumberg & Luke (2011)

Purpose and Methodology The Youth Poll Mail Study (YPMS) conducted a series of data

collection mode comparisons– Goal is to evaluate the feasibility of switching the current RDD

telephone–based Youth Poll to a mail-based survey methodology– Run concurrently with RDD YP to compare coverage, nonresponse, and

key metrics• December 2008• June 2009• December 2009• June 2010


5

Mail YP RDD YP

Age range 16-24 16-24

Survey Interview Method Pencil & Paper CATI

Sampling Single-Stage Stratified Random

List-Assisted Stratified Random

6

Youth Poll Mail Study

Mail YP Frame Coverage June 2010 frame was an updated version of December 2009 frame:

– Aged to remove any youth that were no longer 24 years old– Augmented using additional lists to capture youth that had just turned

16 December 2009 frame coverage

– Comparison with Census estimates of the 16- to 24-year-old population indicated a 95% coverage rate, a 6 percentage point improvement from June 2009 (89%)*

– Coverage of both 16- (67%) and 24-year-old (95%) youth increased substantially since June 2009 (47% and 60%, respectively)

– Coverage of the West region also improved since June 2009


7

June 2009 December 2009

Frame size 33,902,404 36,483,999

Coverage rate 89% 95%

16-year-olds 47% 67%

24-year-olds 60% 95%

West Region 75% 84%

Data Collection Efficiency The Mail YP required fewer contacts to reach the target population

than the RDD YP and did so at less than one-third the cost per completed interview with no additional fielding time required

The Mail YP’s cost per completed survey in June 2010 ($77) was lower than in December 2009 ($99) and roughly equal to that in June 2009 ($76)

– Increase in costs in December 2009 was largely a result of evaluating a push-to-Web solicitation strategy that was considerably less productive


8

Mail YP RDD YP

Length of Time to Field 2.5 months 2.5 months

Contacts 30,000 223,173

Completed Surveys 5,119 4,023

Completion Rate 17% 2%

Cost per Completed Survey $77 $273

Profile of Responders Demographics

– Compared with the RDD YP and census estimates, respondents in the Mail YP were more educated

Telephone Status– Compared with the RDD YP, respondents in Mail YP were:

• More likely to have a cell phone only • More likely to be cell-mostly (i.e., both a landline and a cell phone

who receive all or almost all of their calls on a cell phone)

• Less likely to have a landline only


9

Survey Estimate Comparisons Propensity estimates were statistically identical between the Mail

and RDD YP– True for both overall and Service-specific propensity comparisons

Mail YP tends to provide slightly lower propensity estimates than RDD YP

– Finding is consistent with expectations


10

Key MetricsJune 2010

Mail YP RDD YPGeneral Military Propensity 10% 10%

Army Propensity 6% 6%

Navy Propensity 6% 6%

Marine Corps Propensity 5% 6%

Air Force Propensity 7% 7%

Coast Guard Propensity 4% 5%

11

Item Nonresponse

June 2010 Mail YP Propensity Items2011 ITSEW – Québec, Canada

12

Item Nonresponse In the June 2010 Mail YP, Service-specific propensity grid items

yielded overall missing data rate of 8%– Non-random forms of nonresponse may bias point estimates

Analyzed June 2010 Mail YP data to determine:– Whether there was a pattern to the missing data; and – How refusals should be handled (e.g., multiple imputation, etc.)

Pattern of nonresponse was NOT RANDOM!– Non-propensed youth were more likely to refuse answering all 12

Service-specific items– Conversely, propensed youth more likely to refuse to answer 1-11

Service-specific items, which is probably a result of their preference for a specific Service(s).


13

December 2010 Administration Modified the Service-specific propensity item in the December 2010

questionnaire:

Proportion of missing data reduced by half!– From June 2010 , 8% missing data rates down to 4%


14

15

Incentive Experiment

Combining Prepaid and Promised Incentives

• Mail YP– Periodically conducts experiments to enhance survey quality

• Combining prepaid and promised incentives– Ideally, this would maximize survey quality at a reduced

cost

• Theoretical justification– Trust is the centerpiece of social influence theory (i.e.,

enhancing trust facilitates the persuasion process)– Prepaid incentives may “build trust” such that they might

“magnify” the positive effects of promised incentives

AAPOR 2011 – Phoenix, AZ

16

Methodology• June 2010 Experiment

– Sample of 30,000 young adults ages 16-24 living in US were randomly assigned to 1-of-6 conditions (5,000 each)

• Survey length (long vs. short)• Promised incentive ($0 vs. $5 vs. $10)

– Everyone received $2 cash incentive in the 1st survey invitation package

• Promised incentives sent to respondents that returned questionnaires

– Where possible, all mailing materials sent to youth were identical


17

Results• Nonresponse

– Completion rates for the short survey were higher than for long survey

– Promised incentives help offset completion rate drops in long survey

• Key Metric Measurement– In general, point estimates similar across all the experimental

conditions

• Efficiency– Compared with offering just a prepaid incentive, promised

incentive with long survey reduces costs by approximately 18-26% per completed survey


18

Short;

$0Short;

$5Short;

$10Long;

$0Long;

$5Long;$10

Completed Surveys 17% 19% 22% 12% 15% 18%Enrolled in School 72% 67% 69% 72% 66% 70%Currently Employed 56% 55% 54% 55% 57% 52%General Military Propensity 8% 9% 9% 10% 10% 9%

Note: All estimates in this table are unweighted; 5,000 cases sampled in each condition.

19

Methodology Transition

Summary Several indicators supporting the switch to a mail-based

methodology– Mail YP’s existing frame coverage (95%) of the target population

represented a marked improvement over existing RDD methodologies that do not capture the steadily growing number of cell phone–only young adults

– Metrics compared well across survey modes• Both overall and Service-specific propensity estimates were

statistically identical between the Mail and RDD YP

– Mail YP required fewer contacts than RDD YP to reach the target population

• Mail YP was less than one-third the cost per completed interview

• Mail YP required no additional fielding time

– Mail YP is proving to be a viable vehicle for motivating higher response rates

• Compared with offering no contingent incentive, including a contingent incentive reduces data collection costs by approximately 18-26% per completed survey*


20*Figures are based on a comparison of the cost per completed survey in the $5 contingent incentive condition ($63,745 survey cost ÷ 749 completed surveys = $85.11 per completed survey) and the $10 contingent incentive condition ($68,980 survey cost ÷ 898 completed surveys = $76.82 per completed survey) vs. the $0 contingent incentive condition ($60,000 survey cost ÷ 575 completed surveys = $104.35 per completed survey).

item nonresponse in a mail survey of young adults

Documents