whither survey response rates - tns canadatnscanada.ca/conferences/mria/tnscf-whither survey...

© 2006 MRIA Response Rate Committee

Whither Survey Response Rates

Do They Still Matter?

Gary Halpenny, TNS Canadian FactsDon Ambrose, Consumer Contact

June, 2006

© 2006 MRIA Response Rate Committee

Background

© 2006 MRIA Response Rate Committee

Telephone surveys have been under attack recently on the grounds that:

“Results are no longer accurate nor representative”

Low response rates are cited as the the reason

© 2006 MRIA Response Rate Committee

However: a growing body of research begs to differ

A number of investigative projects in the U.S. have shown:

for most commercial and public opinion applications a 30% response rate produces essentially the same results as a 50% response rate

© 2006 MRIA Response Rate Committee

Some of the research literature:In 1997, two identical surveys, one at 61% response rate and the other at 36%, produced no meaningful differences

This project was replicated in 2003 with 51% and 27% response rates and with similar results

Researchers concluded “carefully conducted polls with relatively low response rates still yield representative samples and accurate data”(Keeter et al, Pew Research)

© 2006 MRIA Response Rate Committee

The reality today is that few commercial telephone surveys even approach the 30% level

The demand for faster turnaround means most telephone response rates are now in the 10% to 20% range

Quick 1 or 2-day polls can yield even lower rates

© 2006 MRIA Response Rate Committee

The Critical Issue!!

Can response rates at these levels still produce accurate and meaningful data?

Clearly more research was needed

© 2006 MRIA Response Rate Committee

MRIA’s Research Project

© 2006 MRIA Response Rate Committee

The Plan

In 2005, the MRIA Response Rate Committeesponsored research to investigate whether response rates as low as 10% can still produce reliable and useful data.

Five Canadian research companies who regularly conduct national omnibus surveys volunteered to combine efforts.

© 2006 MRIA Response Rate Committee

Stage 1:

Using an identical 5 minute question set, each company completed approx. 250 interviews on a single wave of its Omnibus in January, 2006.

1,238 completed interviews in total4 days in field9% aggregate response rate

© 2006 MRIA Response Rate Committee

Stage 2:

Using the same 5 minute question set, each company completed a second sample of approx. 250 interviews over Jan/Feb, 2006.

1,273 completed interviews in total4 to 5 weeks in fieldFirst refusals recontacted31% aggregate response rate

© 2006 MRIA Response Rate Committee

National RDD, age 18+

Weighted to Census for:Age

Gender

Province

Community Size

Both samples were:

© 2006 MRIA Response Rate Committee

Fieldwork undertaken by:

Ipsos-Reid

Maritz

Opinion Search

Synovate

TNS-Canadian Facts

© 2006 MRIA Response Rate Committee

Record of Call Comparison

Higher contact ratio (lower proportion of busy/no answer)

Completion/refusal ratio increases from .26 to .77

Means that fewer good telephone numbers required to yield same number of interviews.

Table on next slide indicates that additional call attempts yield three main benefits

© 2006 MRIA Response Rate Committee

79

1,273

105

1,238

Disqualified

Completed Interviews

1,35231.1%

1,3439.1%

Cooperative Respondents

7801,647

569

5,8434,8262,820

Busy/No AnswerRefusedOther Non-responding

4,348100%

14,832100%

Valid numbers attempted31% RR9% RRDisposition of last attempt

© 2006 MRIA Response Rate Committee

Key Findings

© 2006 MRIA Response Rate Committee

Both Studies Yield Identical Results* for:Incidence of food items used in past 6 mos.List of items bought in last 12 monthsAppliances in HouseholdPrint media readership – not title specificIncidence of Travel outside CanadaPersonal Access to the InternetCell phone ownership and carrier used

*Differences considered significant at the 90% level

© 2006 MRIA Response Rate Committee

9% RR 31%RR Sig. Diff % % Eggs 97 96 N Cold Cereals 86 86 N Cheese (Not Processed) 69 71 N Honey 67 66 N Frozen Pizza 55 55 N

Food Items Used in Past 6 MonthsResults Identical

© 2006 MRIA Response Rate Committee

Items Bought In The Last 12 MonthsSame result regardless of whether category incidence is high, medium or low

9% RR 31%RR Sig. Diff % %

Men’s Or Women’s Clothing 93 93 N Sunscreen/Suntan Lotion 54 54 N Paint or Stain 52 51 N Camping Equipment 23 23 N Car Polish/Wax 21 20 N Traveler’s Cheques 8 8 N

© 2006 MRIA Response Rate Committee

Appliances in HouseholdSimilar findings for both commonplace and more esoteric items

9% RR 31%RR Sig. Diff

% %

Microwave Oven 95 95 N Automatic Dishwasher 63 61 N Gas BBQ 59 57 N Security System 34 37 N Espresso/Cappuccino Maker 14 13 N

© 2006 MRIA Response Rate Committee

Print Media ReadershipSimilar estimates of generic print media consumption

9% RR 31%RR Sig. Diff % %

Read A Daily Newspaper -Yesterday 60 60 N - Past Week 84 84 N

Last Time Read A Magazine - Yesterday 38 40 N - Past Week 72 72 N

© 2006 MRIA Response Rate Committee

Traveled Outside Canada In Past 12 MonthsParallel results for both business and personal travel behaviour

9% RR 31%RR Sig. Diff % % For Personal 32 33 N For Business 8 9 N

© 2006 MRIA Response Rate Committee

Personal Access To The InternetPenetration levels virtually identical

9% RR 31%RR Sig. Diff % %

Any Access 76 76 N

At Home 70 70 N At Work 45 46 N

© 2006 MRIA Response Rate Committee

Cell PhonesNo differences in either ownership incidence or carrier share

9% RR 31%RR Sig. Diff % % %

Has a Cell Phone 58 58 N

Cellular Provider * Bell 29 28 N Telus 25 27 N Rogers 25 25 N Fido 6 6 N Other 11 11 N * Base Total Cell phone owners

© 2006 MRIA Response Rate Committee

Credit Card Ownership and Usage

Higher response rate yields higher incidence of credit card ownership

Among card owners, high RR yields a higher incidence of owning American Express and a lower incidence of MasterCard

Posit that the higher RR captures a more upscale, harder-to-find group of people but not proven in the demos

Equally as likely to be a statistical anomaly

No differences in card used most often

Differences are found here.

© 2006 MRIA Response Rate Committee

Credit Cards Owned 9% RR 31%RR Sig. Diff % % Has Any Credit Cards 78 82 +5

Specific Cards Owned* Visa 67 70 N MasterCard 52 48 -4 American Express 13 18 +5 Diners 1 1 N Any Department Store 45 46 N Any Gasoline Company 15 14 N

Avg. # of Cards Owned* 2.4 2.5 N * Base: Total Credit Card owners

© 2006 MRIA Response Rate Committee

Credit Card Used Most OftenClaimed usage level unaffected by higher response rate

9% RR 31%RR Sig. Diff Base = Owners of Credit Cards % % Visa 49 52 N MasterCard 30 29 N American Express 3 4 N Any Department Store Card 3 3 N Any Gasoline Co, Card 1 1 N

© 2006 MRIA Response Rate Committee

12 Attitudinal Statements Measured

Mean scores the same on 11 attributes out of 12

Difference on the statement related to shopping was statistically significant but would not have changed the interpretation

© 2006 MRIA Response Rate Committee

Attitudinal Statements 9% RR 31%RR Sig. Diff # #

I like to try new and different products 6.0 6.0 N I am willing to pay extra to save time 5.4 5.4 N I lead a fairly busy social life 5.9 6.0 N A person’s career should be their 1st priority 4.9 4.9 N TV is a primary source of entertainment 5.7 5.7 N I have more self-confidence than most people my age 6.9 6.9 N I keep up-to-date with changes in style 5.4 5.3 N I am careful of what I eat 7.2 7.2 N I go out with friends a great deal of the time 5.1 5.2 N To me shopping is a chore rather than a pleasure 6.1 5.9 -0.2 I prefer to postpone a purchase rather than buy on credit 6.6 6.7 N

© 2006 MRIA Response Rate Committee

Conclusions

Previous findings are corroborated – “carefully conducted polls with relatively low response rates still yield representative samples and accurate data”

Important that all other aspects of good survey design also must be present:

The set of telephone numbers is a randomly drawn, representative sample of the universeRespondent selection at HH level is as random as possibleThe data are weighted appropriately

© 2006 MRIA Response Rate Committee

Conclusions (Cont’d)High response rates are still achievable for studies where this is an important design criterion

Fast field turnaround and high response rates are incompatibleAvailable time to complete the fieldwork is the main factorMore focus on the sample management process is required e.g. call scheduling, elapsed time between attempts, etc.

© 2006 MRIA Response Rate Committee

Consider a repeat of this test this fallCan the overall findings be replicated?Are the few data differences found real or merely random data anomalies

Modify the question set somewhatReplace the attitudinal questions with questions related to public policy

Where Next?

© 2006 MRIA Response Rate Committee

Contributors To This Project

Design and AnalysisDon Ambrose - Consumer ContactGary Halpenny - TNS Canadian FactsIvor Thompson - Maritz

Data Merging/TabConsumer Contact

TranslationJL Translation

Census DataSM Research

FieldworkIpsos-ReidMaritzOpinion SearchSynovateTNS Canadian Facts

© 2006 MRIA Response Rate Committee

MRIA Response Rate Committee

Patricia Simmie* CMRP Ipsos Camelford GrahamDon Ambrose CMRP Consumer ContactGary Halpenny CMRP TNS Canadian FactsPeter Chan CMRP TNS Canadian FactsIvor Thompson CMRP Maritz ResearchJohn Smart CMRP SM Research TechnologiesDerrick Gray BBM Canada

*Chair