panel response rates by ted vonk - onderzoekpaleis london 3 october 2006 advances and opportunities...
Post on 11-Jan-2016
213 Views
Preview:
TRANSCRIPT
Panel Response Rates
By
Ted Vonk - Onderzoekpaleis
LONDON 3 October 2006
Advances and opportunities in market research online
Dutch Online Panel Study 2006
COPYRIGHT
This PowerPoint presentation contains the outcomes of the Dutch Online Panel Comparison study 2006 (NOPVO 2006).
It is allowed to publish the outcomes of this study if you refer to:
• The name of the study (NOPVO 2006)
• The authors: Ted Vonk (Onderzoekpaleis), Robert van Ossenbruggen (ProCression) and Pieter Willems (Millward Brown)
Contact: info@onderzoekpaleis.nl
Agenda
• An industry survey: current state of online panel research
• Response in NOPVO
• How important is response?
• Conclusions and recommendations
Gastspreker
Els Molenaar
“Als je wint heb je vrienden”
An industry survey: current state of online panel
research
Some Facts and figures about the Netherlands
Demographics
• Population: 16,3 million
• 18-65 years: 10,5 million
• Internet penetration: 82%
• Broadband penetration: 63% (of all housholds, not internet users)
Market Research
• Total market in 2005: € 278 million (UK: € 1.940 million)
• Spending per capita: € 17 (UK € 32)
• Online data collection: 25% of revenu quantitative (UK 9%)
• Over 25 online panels
Reasons for strong and fast growth of online research
• Growing costs of traditional research because of changing laws (social security fees)
• Decreasing response rates within traditional research
• Strong growth of internet penetration as a result of an active policy of Dutch government
Opinion and market data: over 50% is collected online
• Today, over 50% of research data is collected by online panels
• More than 25 online panels are operational in the Netherlands
• Clients have reacted differently over time
1998 – 2001: SceptisismIs it respresentative? what are the effects of mode?
2002 – 2004: Fully accepted – hardly any quality questionsSpeed and costs seem to have convinced all clients
2005 – now: Again increased quality concernlarge offering, low prices, lack of standards
The Dutch Online Panel Study (NOPVO)
Why?
1. Provide answer to quality concern
2. Provide guidelines for online panel innovation
3. Unique situation: large number of panels available
Research questions NOPVO
1. How do panels work: panel overview
2. Who is the online panel member
3. How do panellists respond?
4. To what extent is opinion and marketing data influenced by panel choice
NOPVO method
1. Make inventory of all online panels in the Netherlands
2. Empirical study
• One survey
• Every panels draws sample of 1000 respondents
• Same fieldwork period of 7 day (no reminders)
19 panels participated:
Representing in totalover 90% of panellists available in NL
NOPVO method
Extensive data available:
Per panel:-Type of incentive-Age of Panel-Invitation policy -etc. Per panel member:
-Panel history-Way of recruitment-Number of invitations-Individual response rate-socio demo, etc.
Per respondent:-Response (time)-Membership of other panels-data of the questionnaire
NOPVO Questionnaire
• Omnibus study to avoid selective response
• Labour and transport
• Political preference
• Judgement of PM and his image
• Religion, going to church
• Moving houses and satisfaction with house and neighbourhood
• Spontaneous and aided awareness of beer-TV brands
• Advertising awareness
• Participation in online research
• Internet usage
• Different types of scaling questions, open ended questions etc.
Participating panels NOPVO
Traditional Full Service part of international chain
Internet start ups Only completed
25 questions of ESOMAR
GfK Benelux
Intomart GfK
Millward Brown Centrum
Synovate
TNS NIPO
Flycatcher
Metrixlab
Multiscope
Netpanel
USP-MC
CIAO
PanelWizard
Trendbox
Traditional Full Service Panel Only
IBT/Team4
Interview/NSS
Motivaction
MarketResponse
RM Interactive
GMI
Lightspeed Research
SSI (Bloomerce)
Panelclix
Interesting facts of the panels
• Majority of panels exists longer than 5 years
• Different ways of recruiting used
• Almost all panels give incentives for cooperation
• Average € 1,20 for 10 minutes
• Most panels have a restriction policy in inviting panel members for a survey
• Overview of all details of all panels available at MOAWEB.NL
• Several panels do not have full details about interview history of panel members
Panel response
19.000 panel members invited….
9.462 completed the questionnaire…..
Response of 50%
75% of response is realised within 24 hours
Time Thursday Friday Saturday Sunday Monday Tuesday Wednesday Total
01.00 - 07.00 0% 4% 4% 2% 1% 4% 5% 2%
07.00 - 09.00 0% 9% 5% 3% 5% 6% 4% 4%
09.00 - 12.00 0% 23% 20% 21% 22% 17% 17% 11%
12.00 - 14.00 1% 11% 13% 17% 11% 12% 13% 7%
14.00 - 17.00 34% 17% 20% 17% 17% 13% 19% 26%
17.00 - 20.00 28% 16% 16% 17% 18% 16% 14% 22%
20.00 - 23.00 32% 13% 13% 18% 21% 21% 24% 23%
23.00 - 01.00 5% 6% 8% 5% 5% 12% 4% 6%
Total 50% 27% 8% 6% 5% 3% 1% 100%
Substantial differences in response per panel
Level of response is a result of several factors
• Panel age
• Way of recruitment
• Incentives
• Panel care
• Etc.
It is hard to determine the effects of a single factor
Response per panel
18 19
36 39 39 39 40 4149
54 55 56 60 60 62 66 68 7077
50%
Reasons for high/low response (1)
• Panels with a high response rate did not only invite their best responding panel members …
Panels with a high response rate are mainly new panels
• Panels with a low response rate are panels that do not clean up their panel database
• Panels invited panel members that did not complete one questionnaire in over a year despite many invitations
• Number of panel members is an important communication topic to the market
Reasons for high/low response(2)
• Panel age:
• Composition of samples to panel age differs a lot per panel
• New panel members respond better than old panel members
• Method of recruitment:
• Recruitment via traditional research (-instruments) gives best response (are willing already)
• Self registration or via links/banners gives significant lower response
• Incentives
• Consistency of incentive is important
• Positive effect of incentive stops at 1 euro
Composition of samples to panel age differs a lot per panel
Panel age in gross sample per panel
97
75 74
4841 37 35 32
2013 11 8 8 5 5 4 2
30
3
15 12
69
60
517
80
6 10 11 11 175 4 11
16
10 14
46 5060
51
81 79 82 81 7989 92 86
53
Panel
%
<6 months 6-12 months > 1 year
New panel members respond better than old panel members
Response and duration of panel membership
67%
61%
56%
54%
47%
47%
0 months
1-2 months
3-4 months
5-6 months
6-12 months
> 1 year
respons %
Recruitment methods vary substantially across panels
Recruitment method of panel members in gross sample
16%
15%
22%
6%
18%
1%
1%
5%
10%
5%
Buying addresses
Self registration
Via links/banners
Snowballing
By telephone
From existing panels
Advertising in print media
Via website surveys
Via traditional research
Other
Most panels use a mixture of recruitment methods
Recruitment method per panel
100
61 56
31
142
22
2 9
1814
23 18
15
36
25
62 7416
5
13
11
81
2
18
27
1
27
11
43
15
10
5
35
6
37
17
35
19
64
18
100 100%
Other
Via traditional research
By telephone
Buying addresses
Self registration
Via links/banners
Self registration or recruitment via links/banners gives significant lower response
Response per recruitment method
73%
66%
61%
57%
53%
51%
39%
39%
Existing panel
Snowballing
Via traditional research
Buying addresses
By telephone
Via website surveys
Self registration
Via links/banners
respons %
Certainty about level of incentive is important
Response per type of incentive
70%
62%
58%
49%
39%
29%
Cash
No incentive
Value points
Charity and lotteries
Value points and lotteries
Lotteries
response %
Positive effect of incentive stops at 1 euro
Response and value of incentive
59%
57%
37%
42%
> 1 euro
1 euro
< 1 euro
No (fixed) incentive
response %
Panel members love to complete questionnaires……
Reasons to participate in online research
43
38
34
21
20
26
22
17
9
8
I love completing questionnaires
I love to express myself
Because of the incentives
Curiosity
Interest in new products/ideas
%
Total reasons Most important reason
Incentives drives young people
"Because of the incentives" per age category
28
22
15
10
17
18 - 24 year
25 - 34 year
35 - 49 year
50 - 65 year
Total
%
Some other results
• Heavy Internet users respond quicker
• Socio demographics
• Women respond better than men
• Older people respond better than younger people
• Average completion time 12,5 minutes
• Fastest panel: 11,1 minutes
• Slowest panel: 15,5 minutes
• Better respondents respond ……better
Better responding panel members respond …… better
• Average response rate of individual panel members during last 12 months calculated with data about invitations and completes from panel
Average respons during last 12 months per DOPS respons category
67%
54%
31%
Complete
Stopped
Non respons
%
Is a high response rate important?
Research outcomes do not depend on response percentage
0
5
10
15
20
25
30
35
H L min max
# w
ord
s
number of words typed in open answer
Research outcomes do not depend on response percentage
voting in upcoming national elections
0
0,05
0,1
0,15
0,2
0,25
0,3
0,35
0,4
CD
A
Chr
iste
nUni
e
D66
Gro
enLi
nks
Gro
ep W
ilder
s
LPF
(Li
jst
Pim
For
tuyn
)
Pvd
A
Par
tij v
an d
e D
iere
n
SG
P
SP
(S
ocia
listis
che
part
ij)
VV
D
H
L
Research outcomes do not depend on response percentage
awareness of beer brands
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
De
Kon
inck
Bud
wei
ser
Car
lsbe
rg
Leff
e
Cor
ona
Mae
s
Mur
phys
Ste
lla A
rtoi
s
Alfa
H
L
min
max
Research outcomes do not depend on response percentage
number of words typed in open answer
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
Working Housekeeping Non employed Unable to work Retired Student other
H
L
Research outcomes do not depend on response rate
• Low versus high responding panels
• No different means on survey questions
No response effects explained
• Non-response effects only occur if there are differences between respondents and non-respondents
• Non-response effects take place while recruiting panels
• double opt-in
• explicit consent to participate in research
• So, the relevant non-response group has been filtered out before panellists are invited for surveys
• Potential risk for non-response effects in panels:
• when response relates to survey topic (selective response)
Reponse rate: panel strategy
• A high response rate does NOT indicate a high sample quality
• Response rate is a panel strategy component, driven by rules of economics
• panel setup and maintenance determine response rate
• the value of response rate is evaluated against recruitment costs
Conclusions
Conclusions
• With online panels, only a specific group of respondents is interviewed.
• Selection takes places during recruitment, not with inviting panellist for an individual study
• Therefore, response rate does not reflect sample quality: it reflects a panel business strategy
• This self selection issue is not new to market research: we’ve seen the same with CATI and P&P
• Challenges for online panel research are to reach the other group of people
Recommendations
• Deploy other, new recruitment methods
• Remove badly responding panellists
• Use online panels for the right kind of research goals
use panels for
• continuous research or trend measurement
• concept testing
but not for
• incidence rating
• Do not swap panels in a continuous research project
• For 0 versus 1 measurement studies, match sample on panel historical data
Thank you
tedvonk@onderzoekpaleis.nl
pieter.willems@nl.millwardbrown.com
top related