testing for coverage bias when combining directory-listed and cellphone samples t. m. guterbock, a....

Testing for Coverage Bias when Combining Directory-Listed And Cellphone Testing for Coverage Bias when Combining Directory-Listed And Cellphone SamplesSamples

T. M. Guterbock, A. Diop, J. M. Ellis, J. L. P. Holmes and K. T. Le, Center for Survey Research, University of T. M. Guterbock, A. Diop, J. M. Ellis, J. L. P. Holmes and K. T. Le, Center for Survey Research, University of VirginiaVirginia

RDD+Cell costs too much!

Cell-only households are increasing.

Researchers are using a dual-frame sample design combining RDD and cell phone frames (“RDD+Cell”).

RDD+Cell costs more than RDD because cell phone interview costs are 2 to 3 times higher than RDD costs.

Random digit dial (RDD) continues to lose efficiency due to lower phone number densities in sampled call groups.

RDD is especially inefficient in local surveys, where households must be screened for location.

Do we really need all that RDD?Cellphone samples today are used to capture underrepresented groups that RDD samples were originally designed to capture.

Modify the RDD+Cell design to use Electonic White Pages sample. The “EWP+Cell” design covers all phone HH except unlisted landline-only.

EWP+CellBenefits DrawbacksLower cost Noncoverage biasGeographic specificityFaster field time

Are noncoverage biases small enough and benefits large enough to support using EWP+Cell?

We showed (AAPOR 2008) that in the 2006 National Health Interview Survey, simulated results from EWP+Cell were very close to RDD+Cell results.

Three “citizen satisfaction” surveys conducted in 2008, in three suburban Virginia counties

Each survey used a triple-frame design: RDD, EWP and cellphone frames

Surveys were similar in content, length and design Percentage of the telephone universe that has cell

phones was estimated separately for each countyFinal data were weighted for estimated cellphone-

only prevalence in each county$10 incentives for cellphones ($5/$10 test in PWC)Triple frame design allows direct test:

RDD+Cell vs. EWP+Cell

www.virginia.edu/surveys

AAPOR Annual Conference

Hollywood, FL, May 2009

But NHIS is a very large , national in-person But NHIS is a very large , national in-person survey. Will these results hold up in smaller survey. Will these results hold up in smaller

local surveys conducted by telephone?local surveys conducted by telephone?

1

3

1

2 2

8

5

7 7

3

1 1

0

1

2

3

4

5

6

7

8

9

-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7

Number

of

Items

Raw Percentage Point Difference: EWP+Cell - RDD+Cell (41 items)

Albemarle County 2008Count of Survey Items by Percentage Point Difference

12

1

8

14

24

13

10

12

1

0

5

10

15

20

25

30

-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7

Number

of

Items

Raw Percentage Point Difference: EWP+Cell - RDD+Cell (77 items)

Chesterfield County 2008Count of Survey Items by Percentage Point Difference

2 2 2

11

17

43

21

0

2

4

6

8

10

12

14

16

18

-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7

Number

of

Items

Raw Percentage Point Difference: EWP+Cell - RDD+Cell (44 items)

Prince William County 2008Count of Survey Items by Percentage Point Difference

RDD+Cell covers all five telephone segments

EWP+Cell covers four of five segments

One example: For this item, the differences in responses were small and within the margins of error for the surveys. Ratings were dichotomized for analysis.

Items with sufficient n sizes were compared across EWP+Cell and RDD+Cell within each survey. Raw percentage differences were rounded and tallied. Most differences are smaller than the survey sampling errors.

Prince William (DC suburbs)

Albemarle (C’ville suburbs)

Chesterfield (Richmond suburbs)

N CPH N CPH N CPH RDD 623 0.82 278 1.12 637 1.16 EWP 693 1.09 334 1.39 980 1.40

CELL 187 0.56 155 0.57 157 0.40 Combined samples 1503 767 1774

Estimated % cell phone only: 19.0% 8.4% 8.0% % of unlisted in RDD: 23.8% 10.1% 17.6%

% unlisted landline only: 2.2% 1.4% 1.3% MOE (weighted triple-frame) ±2.9 ±3.8 ±2.7

County % satisfied with overall county government services

Δ RDD EWP CELL RDD+CELL EWP+CELL

Prince William 89.4 89.3 88.0 89.3 89.4 0.1% Albemarle 90.5 92.6 94.5 92.0 93.5 1.5% Chesterfield 85.5 86.9 85.4 85.5 86.6 1.2%

Assume $33 per interviewing hour CPH Cost Total cost Prince William

EWP+Cell EWP (n=800) 1.09 $24,220 $38,206

save 17% Cell (n=200) 0.56 $13,986 Prince William

RDD+Cell RDD (n=800) 0.82 $32,195

$46,091 Cell (n=200) 0.56 $13,986

Albemarle EWP+Cell

EWP (n=800) 1.39 $18,993 $32,772 save 12% Cell (n=200) 0.57 $13,779

Albemarle RDD+Cell

RDD (n=800) 1.12 $23,571 $37,350

Cell (n=200) 0.57 $13,779 Chesterfield

EWP+Cell EWP (n=800) 1.40 $18,857 $37,557

save 9% Cell (n=200) 0.40 $18,700 Chesterfield

RDD+Cell RDD (n=800) 1.16 $22,759

$41,459 Cell (n=200) 0.40 $18,700

Note: Cellphone costs include $10 incentive and $1 processing

2CELL + ULL

17.7%

4CELL + LLL

34.3%5

LLL ONLY17.2%

3ULL ONLY

14.2%1CELL ONLY

16.6%

Cell

Overlap

EWP

1CELL ONLY

16.6%

2CELL + ULL

17.7%

4CELL + LLL

34.3%

3ULL ONLY

14.2%

5LLL ONLY

17.2%“ULL”=Unlisted

Landline

“LLL”=Listed Landline

““CPH” = Completions per hourCPH” = Completions per hour

Prince William County 2008

1CELL ONLY

16.0%

2CELL + ULL

24.2%

3ULL ONLY

2.2%

4CELL + LLL

54.6%

5LLL ONLY

3.0%

Cell

Overlap

EWP

Albemarle County 2008

1CELL ONLY

8.9%

2CELL + ULL

11.3%3

ULL ONLY1.4%

4CELL + LLL

71.2%

5LLL ONLY

7.2%

Cell

Overlap

EWP

Chesterfield County 2008

1CELL ONLY

8.6%

2CELL + ULL

16.0%

3ULL ONLY

1.3%

4CELL + LLL

68.8%

5LLL ONLY

5.3%

Cell

Overlap

EWP