dealing with the effects of panelist experience - c.ymcdn.com · proprietary and confidential 7...

AUTOMOTIVE

BEAUTY

COMMERCIAL TECHNOLOGY

CONSUMER TECHNOLOGY

ENTERTAINMENT

FASHION

FOOD & BEVERAGE

FOODSERVICE

HOME

OFFICE SUPPLIES

SOFTWARE

SPORTS

TOYS

WIRELESS

1

Dealing with the Effects of Panelist Experience

Inna Burdein, Ph. D.

Director of Panel Analytics

Copyright 2008. The NPD Group, Inc. All Rights Reserved. This presentation is Proprietary and Confidential and

may not be disclosed in any manner, in whole or in part, to any third party without the express written consent of NPD.

Proprietary and Confidential

2

Background

Anyone browsing the Internet is bound to come across numerous survey offers

Those looking for surveys to take will have endless options

Google “Paid Surveys,” and within seconds you have access to sites that link you to numerous online panels


3


4


5


6

Background


7

Point being…

Anyone looking to take surveys can become a member of numerous online panels within minutes

In turn, a segment of the population has emerged that belongs to numerous panels and takes surveys regularly

Even those belonging to one or two panels may have been on those panels for years

It’s no surprise that “Professional Respondents” has been a hot topic in this industry


8

Are Professional Respondents Saints or Sinners?

Sinners

– Survey gamers looking for incentives at little cost

– Speed through surveys, opt out of responding

– Some research suggests experienced panelists underreport

Saints

– Cost efficient panelists that enjoy taking surveys

– They like to have their opinions heard and answer many surveys

– Some research revealed no difference between experienced and non-experienced panelists

– Some research found more accurate reporting among the experienced


9

Questions

① What is really behind the “professional” effect and why are the findings so inconsistent?

② Is the experience effect consistent across surveys?

③ Is the experience effect consistent across panelists?

④ What is the impact when left untreated?

⑤ How can we mitigate its effects?


10

① What is the effect of experience?

Existing research varies in several critical ways:

– How experience is operationalized

• Surveys sent, taken

• Tenure (i.e. time on panel)

• Number of panel memberships

– The dependent variable

• Subject matter

– Behavior (e.g. Do you watch TV?)

– Extent of behavior (e.g. How many hours do you watch TV?)

– Attitudes (e.g. Do you enjoy watching TV?)

• Question formats

– “Select all that apply”

– Likert scales


11

Stepping back: Theory of Reporting

Why might someone “experienced” report less?

– Professional respondent wants to end the survey quickly in order to gain the incentives at less “cost”

– Lack of enthusiasm. The survey is no longer novel and hence the interest and motivation to be accurate has waned

– In the case of Screeners that lead into additional surveys, an experienced panelist may want to avoid the additional work

– Satisficing - offer the minimal amount of information necessary but still feel like you contributed.

Experience effect should surface when surveys become

familiar and predictable


12

Available Data

The NPD Group is in a unique position to study the impact of experience

– Bulk of our work involves repeated Purchase Tracking Surveys (i.e. “Trackers”)

• Trackers begin with a “Screener” or a set of questions that based on response lead to additional surveys.

– Panelists are exposed to similar surveys over time that follow a predicable pattern (i.e. learning opportunity)


13

Impact of Experience

0

0.5

1

1.5

2

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

trackers taken

ca

teg

ori

es

se

lec

ted

Trackers taken > Trackers sent, Surveys taken/sent, Panel Membership, Tenure


14

② Is the experience effect consistent across surveys?

What survey qualities may impact reporting?

– Subject matter

• Relevant, exciting topics may cause over-reporting

– Length of task

• A long list may cause skimming; list after list may cause opting out

– Format

• “Select all that apply” will be more sensitive to reporting than a single opinion poll


15

Impact of Experience on Three Different Screeners

0%

5%

10%

15%

20%

25%

30%

0 2 4 6 8 10 12 14 16 18 20 22 24

Trackers Taken

Perc

en

t o

f C

ate

go

ries

Sele

cte

d

Screener 1 Screener 2 Screener 3


16

Impact of Experience on Four Different Questions

0%

10%

20%

30%

40%

50%

0 2 4 6 8 10 12 14 16 18 20 22 24

Trackers Taken

Per

cen

t o

f C

ateg

ori

es

Sel

ecte

d

Types of music you listen to

Digital activities you do

Video game systems you've played

Ways you've watched a movie


17

③ Is the experience effect consistent across panelists?

What panelists’ qualities may impact reporting?

– Demographic

• Women may be more excited to share their clothing purchases, resulting in more over-reporting than men

– Motivation

• Someone concerned with the incentives, with no respect for the integrity of the research, may be more likely to opt out

– Personality

• An honest “Type A” may not exhibit an experience effect because they are always responding with utmost accuracy


18

Impact of Experience by Age and Gender

0%

5%

10%

15%

20%

25%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Trackers Taken

Pe

rce

nt

of

Ca

teg

ori

es

Se

lec

ted

Younger Men Older Men Younger Women Older Women


19

Impact of Experience by Question and Gender

0%

5%

10%

15%

20%

Ga

me

Syste

ms

Pla

ye

d

Dig

ita

l

Activitie

s

Do

ne

Va

rie

ty o

f

Ca

teg

ori

es

Pu

rch

ase

d

Nu

mb

er

of

Sh

op

pin

g

Tri

ps

Fa

sh

ion

Ca

teg

ori

es

Pu

rch

ase

d

Sh

ift

in R

ep

ort

ing

men women


20

1. A/B Panelists

Make few errors, enjoy surveys

2. C Panelists

Have some inconsistent responses, struggle with difficult questions

3. D Panelists

Have some straightlining, some opting out, don’t enjoy surveys

4. F Panelists

Exhibit a lot of fraudulent behavior, from inconsistent reporting to opting out

Fraud Groups


21

Impact of Experience by Fraud Groups


④ What is the Impact if Left Untreated?

NEW EXPERIENCED

WOMEN 11% 5%

MEN 8% 3%

22

Assume the Following Incidence for Purchasing X:


23

Incidence after Weighting on Gender Only

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Proportion of New Panelists

Allowing Percent of New Panelists in sample to shift


24

Incidence after Weighting on Gender and Experience

Allowing Percent of New Men to Shift

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

0% 10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Proportion of Men within New Panelists


25

It is about the Interactions

New Panelists are unlike Experienced Panelists

– New Panelists vary month to month due to recruitment strategies

• Campaign to recruit more young men

• An affiliate source that brings in F panelists

– Experienced panelists tend to represent:

• Smaller households

• Less children in household

• Older demographic

• All demographics that lead to greater responsiveness

} All these shifts will

impact reporting

If you just treat the main effect of experience, and not

the interaction effects, you will fix some of the variance


26

⑤ Mitigating the Experience Effect

Retroactive Options

– Pull Sample on Experience

– Weight Sample on Experience

– Adjust the Reporting by Experience

Proactive Options

– Push for Accuracy

– Reduce Fraud

– Verify Responses


Pulling on Experience

Pull Sample on Trackers Taken

– Ensuring a finer control earlier in one’s career

• 0 trackers

• 1 tracker

• 2 trackers

• 3-4 trackers

• 5+ trackers

– Figure out which demographics have shown the greatest experience impact and nest them within the trackers

Pro: Straightforward approach

Con: Capacity concerns, underutilizing new recruits


28

Projecting on Experience

Project Sample on Trackers Taken

– Ensuring a finer control earlier in one’s career

• 0 -1 tracker

• 2 - 4 trackers

• 5+ trackers

– Figure out which demographics have shown the greatest interaction effect with experience, and nest them within the trackers

Pro: Allows less restriction when pulling sample

Con: May result in difficult to reach targets, causing more variance


29

Rather than altering your sample, adjust the reporting

Adjust the Dependent Variable by Trackers Taken

– Brand new panelists’ categories will be diminished by .5

– Highly experienced panelists’ categories will be increased by 2

Pros: No sampling restrictions, avoid adding variance to variables that don’t have an experience effect

Cons: Complexity of figuring out where to draw that line, and which variables to adjust

0

0.5

1

1.5

2

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

trackers taken

ca

teg

ori

es

se

lec

ted


30

Building a Model of Reporting

EXPERIENCE

– Trackers taken

– Quality of experience

• Qualifying for a node

• Time it took to complete

• How much person liked/disliked the survey

PERSONALITY

– Motivation for taking the survey

– Type of respondent

• Fraudulent

• Excited

• Honest

INTERACTIONS between the two


31

Proactive Ideas

Push for Accuracy

– Quality Pledge: “Receiving your honest and thoughtful answers is vital to the integrity of the research that we do”

Reduce Fraud

– Identify “F” Panelists and remove them from panel

– Point out discrepancy in reporting

Verify Response

– If you suspect over-reporting: “Was any of this actually bought earlier than this week?”

– If you suspect under-reporting: “Was there anything you left out?”


Summary

Weighting on demographics is accepted as a must to ensure unbiased and consistent estimates.

Experience contributes greater discrepancies than skews in age and gender

The focus on “professional respondents,” multi panel membership has drawn attention away from the brand new panelists

– Those “new” to the panel and “new” to the specific survey experience will bias your estimates most

As online samples become more diverse, with river sampling and social network sampling on the rise, it will be even more critical to look past demographics, and consider the experiences and the motivations of these new recruits.

32

AUTOMOTIVE

BEAUTY

COMMERCIAL TECHNOLOGY

CONSUMER TECHNOLOGY

ENTERTAINMENT

FASHION

FOOD & BEVERAGE

FOODSERVICE

HOME

OFFICE SUPPLIES

SOFTWARE

SPORTS

TOYS

WIRELESS

33

Dealing with the Effects of Panelist Experience

Inna Burdein, Ph. D.

Director of Panel Analytics

Copyright 2008. The NPD Group, Inc. All Rights Reserved. This presentation is Proprietary and Confidential and

may not be disclosed in any manner, in whole or in part, to any third party without the express written consent of NPD.

dealing with the effects of panelist experience - c.ymcdn.com · proprietary and confidential 7...

Documents