panel duplication and recruitment: esomar panel conference orlando 2007

1ESOMAR Panel Research Conference Orlando 28-30 October 2007

DUPLICATION & MULTI-SOURCE ONLINE PANEL RECRUITMENTReal quality differences or idle rumours?

Kristof De WulfSam Berteloot


Content

Some major concerns ...

Research model & hypotheses

Effects of multiple membership

Effects of recruitment

Discussion & implications



“1% of panel members accounts for 34% of questionnaires completed”(CASRO 2006)

“It's been difficult to duplicate the results from one panel "so that people in the business can have confidence that the results they're getting are

valid and predictable”(ClickZ Network 2007)

“The market research industry faces a crisis of ‘over-fishing’ and poor respondent quality” that demands solutions to “improve the quality of

online samples and associated responses”(comScore 2004)

“In the past year, marketers and researchers alike have focused unprecedented attention on the accuracy of research – specifically, data

from online panels, which can produce wildly varying results for the same questions within and across suppliers”(American Marketing Association 2007)

“We need to stop this blind trust at the client side. And the industry needs to help clients understand what quality means, and why data quality needs

to be a key differentiator”(Research World April 2007)


Content








Telephone recruitmentRandom digit dialingSpecialised telemarketing field agency209 new panel members recruited


Street, event, mall intercept recruitmentFace-to-face recruitment Several events & locationsWeekdays as well as during the day199 new panel members recruited


E-mail recruitmentPermesso.be opt-in e-mail list Subscribers receive targeted offers and discounts for various products and services195 new panel members recruited


Online advertising recruitmentOnline recruitment campaign via TradeDoublerTargeted campaign 253 new panel members recruited


Pop-up recruitment after on-site surveyNetwork of 155 websitesPop-up asking website visitors for their opinion on the website with opt-in at the end386 new panel members recruited


Response status Click rate Participation rate Drop-out rate Fill-out time


Response quality Item response consistency Response styles Data structure equivalency Open answer detail Self-reported quality of responses


Panel member feedback Motivation to subscribe Survey satisfaction Panel member satisfaction


Multiple online panel membership

Yes No


Yes No



6 data sources to test effects of type of recruitment vehicle (Belgian panel only)− Data source 1: subscription database

Small survey containing questions about their socio-demographic and internet profile.− Data source 2: business-to-business study (wave 1)

Questionnaire about their professional profile 48 hours after subscription.− Data source 3: indirect quality test (wave 2)

Fake packaging test was developed: respondents were asked to give their opinion about the interface of two car radios. They also provided information about their socio-demographic profile and motivation to subscribe.

− Data source 4: category involvement and adoption study (wave 3)Questions about media profile, consumption of FMCG goods, technology adoption, brand sensitivity, social class, values, …

− Data source 5: brand tracking study (wave 4)Study assessing brand awareness, advertising recall, … for a brand of liquor.

− Data source 6: panel member satisfaction study (wave 5)Panel members were asked about their satisfaction with the XL Online Panels panel: the recruitment procedure, invitations, the way of asking questions, the incentive, and so on. Also data on quality of completion, loyalty, motivation to participate to surveys and socio-demographic information were collected.

1 data source to test effects of multiple online panel membership (Belgian panel only)− Data source : panel member satisfaction study

Panel members were asked about their satisfaction with the XL Online Panels panel: the recruitment procedure, invitations, the way of asking questions, the incentive, and so on. Also data on quality of completion, loyalty, motivation to participate to surveys and socio-demographic information were collected.


Content







0% 10% 20% 30% 40% 50% 60%

Lower or secondary education

Non-university continued education

(Post) university education

2.500 euro or less

2.500 - 3.500 euro

More than 3.500 euro

Full-time professionally active

Part-time professionally active

Unemployed / looking for a job

Retired

Houseman / housewife

Student

Disabled

Multiple panels One panel

Who are people being subscribed to multiple panels?

People subscribed to multiple panels do not differ from people subscribed to one panel only in terms of education, professional situation, and monthly net family income.

People subscribed to multiple panels do not differ from people subscribed to one panel only in terms of education, professional situation, and monthly net family income.

Significant difference between groups (95% confidence interval)

Professional situation

Monthly net family income

Highest education


Who are people being subscribed to multiple panels?

People subscribed to multiple panels are more internet savvy: they have been using the medium longer and use it more often and intensively.

People subscribed to multiple panels are more internet savvy: they have been using the medium longer and use it more often and intensively.


0% 10% 20% 30% 40% 50% 60% 70%

No idea

Before 2000

Before 2003

After 2003

Less than 5 days per week

5 days per week

6 days per week

7 days per week

Less than 15 minutes

15-29 minutes

30-44 minutes

45-59 minutes

1-2 hours

2-3 hours

More than 3 hours


Daily usage of internet (hours)

Number of days using

internet

Usage history internet




















Yes No


Yes No


0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Overall click rate

Drop-out rate

Participation rate



No difference in response status: while the overall click rate is higher among subscribers to multiple panels, drop-out rates are higher, resulting in a comparable participation rate.

No difference in response status: while the overall click rate is higher among subscribers to multiple panels, drop-out rates are higher, resulting in a comparable participation rate.





















Yes No


Yes No



Response quality is slightly better among subscribers to multiple panels: they indicate to fill out surveys more truthfully, to be better concentrated when answering questions, and to try harder while filling out surveys.

Response quality is slightly better among subscribers to multiple panels: they indicate to fill out surveys more truthfully, to be better concentrated when answering questions, and to try harder while filling out surveys.


0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 5

I always put in a lot of energy filling out the survey

I always try my very best while filling out the survey

I am always very concentrated when I answer to the questions

I always think well before answering to the questions

The quality of my answers is always very high

It just fill out something as I am only interested in participating in the contest

I typically fill out the survey truthfully


Self-reported quality of responses (5-pt

scale)



Response quality is better among subscribers to multiple panels: item response consistency between surveys is higher.Response quality is better among subscribers to multiple panels: item response consistency between surveys is higher.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Year of birth (between surveys)

Highest diploma obtained (between surveys)


Item response consistency





















Yes No


Yes No



Panel member feedback is more positive among subscribers to multiple panels: overall satisfaction, intention to participate, positive word-of-mouth and recommendation to others are higher.

Panel member feedback is more positive among subscribers to multiple panels: overall satisfaction, intention to participate, positive word-of-mouth and recommendation to others are higher.

0 1 2 3 4 5 6 7 8

Overall satisfaction (10-pnt scale)

Recommendation to others (5-pnt scale)

Positive word-of-mouth (5-pnt scale)

Intention to participate (5-pnt scale)



Panel member satisfaction &

intentions



Intrinsic motivations to learn new things are higher among subscribers to multiple panels. Extrinsic motivations (receiving rewards) are less important and not different between both groups.

Intrinsic motivations to learn new things are higher among subscribers to multiple panels. Extrinsic motivations (receiving rewards) are less important and not different between both groups.


0% 5% 10% 15% 20% 25% 30% 35%

I am interested in receiving a reward

I want to share my opinion

I want to help others

I like doing this

I want to learn new things


Reasons to subscribe


Content

























Yes No


Yes No



Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept. There are no significant differences in fill-out time between modes of recruitment.

Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept. There are no significant differences in fill-out time between modes of recruitment.


0% 10% 20% 30% 40% 50% 60% 70%

Click rate

Drop-out rate

Participation rate

Fill-out time (% of 1 hour)

Pop-up survey Online ad E-mail Intercept Telephone

Average across 5 data collection waves



Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept.Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept.

0%

10%

20%

30%

40%

50%

60%

70%

wave 1 wave 2 wave 3 wave 4 wave 5

E-mail

Online ad

Telephone

Pop-up survey

Intercept

Participation rate per data collection wave





















Yes No


Yes No


0% 10% 20% 30% 40% 50% 60%

Non-response open answers

Midpoint response style

Extreme response style

Acquiescence response style

Disacquiesence response style



Response styles are relatively more present among people recruited via online ads. Non-response to open ended questions is larger among people being recruited via e-mail.

Response styles are relatively more present among people recruited via online ads. Non-response to open ended questions is larger among people being recruited via e-mail.


Response style & open answer detail based on wave 3



Item response consistency is comparable between modes of recruitment. Item response consistency is comparable between modes of recruitment.


0% 20% 40% 60% 80% 100% 120%

Age (between surveys)

Gender (between surveys)

Having children (within survey)


Item response consistency based on

wave 3



Self-reported response quality is comparable between modes of recruitment.Self-reported response quality is comparable between modes of recruitment.


0,00 0,50 1,00 1,50 2,00 2,50 3,00 3,50 4,00 4,50 5,00

I always put in a lot of energy filling out the survey

I always try my very best while filling out the survey

I am always very concentrated when I answer to the questions

I always think well before answering to the questions

The quality of my answers is always very high

It just fill out something as I am only interested in participating in the contest

I typically fill out the survey truthfully


Self-reported quality of responses (5-pt

scale) based on wave 5




















Yes No


Yes No



People recruited via online ads are most driven by intrinsic motivations: providing their opinion and liking to fill out surveys. People recruited via offline methods are most driven by ‘helping others’ (researchers and charity).

People recruited via online ads are most driven by intrinsic motivations: providing their opinion and liking to fill out surveys. People recruited via offline methods are most driven by ‘helping others’ (researchers and charity).


0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

I can help the researchers

I want to provide my opinion

I want to qualify for winning a prize

I can support charity

I want to learn more

I like filling out surveys


Reasons to subscribe based on wave 5 (5-

pt scale)



Panel member satisfaction among people recruited via online ads is significantly higher.Panel member satisfaction among people recruited via online ads is significantly higher.


- 1,00 2,00 3,00 4,00 5,00 6,00 7,00 8,00 9,00

Survey length satisfaction

Survey lay-out satisfaction

Survey questioning satisfaction

Survey topic satisfaction

General project satisfaction

General panel member satisfaction


Project satisfaction based on waves 4 and 5 (5-pt scale)

Panel member satisfaction based on wave 5 (5-pt scale)


Content








1. We should not be too concerned about the fact that people join multiple panels as their quality is not inferior to people who have only joined one panel While they are more heavy users of the internet, they are positive towards

the panel they have joined, reveal better click rates, show slightly better response quality, and attach a greater importance to extrinsic as well as intrinsic motivations

2. Our results also indicate that we should abandon stereotyping people being subscribed to multiple panels As opposed to what many think, they are not lower social class, weakly

educated housewives filling out surveys as a complementary source of funding

3. Multi-method recruitment for building online panels is not a necessary condition for building a good quality panel Most recruitment methods have shown to recruit quite similar profiles Online recruitment methods have proven to generate comparable results

in terms of response status and slightly better results in terms of response quality than offline recruitment methods

Given that online recruitment is much more cost-effective, online recruitment method not only makes sense from a return-on-investment perspective, but also from a quality perspective


Limitations & directions for future research

1. Can the same results be replicated across more markets? All research hypotheses were tested based on panel members from

Belgium only What is the impact of the following factors on our results? Economic country situation Internet penetration within country Maturity of internet research in the country Cultural differences between countries …

2. Can the same results be replicated over longer timeframe? 5 consecutive research studies were included as observations More long-term research is needed in order to check whether the effects

remain the same.

3. Can the same results be replicated in other panels / across other recruitment channels? Multiple panel membership What is the effect of the number of memberships? What is the effect of the type of membership?

Type of recruitment source What are differences between different e-mail lists? How can online advertising be optimized (creative execution, publishers used, …)? …


THANK YOU

More information?

Kristof De Wulf: [email protected] Berteloot: [email protected]

panel duplication and recruitment: esomar panel conference orlando 2007

Business