arf foq2 day router presentation

FOQ2: Routing Insights and Recommended Practices

Nancy Brigham, Ph.D.

Global Head of Research-on-Research Ipsos Interactive Services

Chuck Miller

President Digital Marketing & Measurement (DM2)

Agenda & Team

• Overview: 15 minutes

• Insights: 30 minutes

The Team:

• Co-chairs: Michael Fallig (Cogenti) & Chuck Miller (DM2)

• Quantitative Design: Nancy Brigham (Ipsos) group leader, Scott Porter (Added Value), Paul Johnson (SSI), Mike Conklin (GfK), Robin Murphy (Ipsos), John Bremer (Toluna)

• Qualitative Design: Paul Johnson (SSI), Efrain Ribeiro (Lightspeed), Steve Gittelman (MKTG Inc.)

• Support Deliverables: Peter Milla (Peter Milla Consulting)

What is Routing?

• Routing is a method of assigning potential respondents to a set of surveys that need sample

Just as we do with panels, but this is done in real-time in an automated manner

• The technology used to do this is a router

Routers use software and business rules to assign a stream of potential respondents to available surveys

Traditional vs. Routed Approach

Screener Study 8

Outcome: NQ

Outcome: QUALIFY

Completes

Study 8

DONE; No

study

completed

…and so on for each individual study

NQ = Not qualified (e.g., ineligible, over quota, failed security screen, study closed, etc.)

Screener 42

Outcome: NQ

Screener 23

Outcome: NQ

Screener 71

Outcome:

QUALIFIED

Enters and

Completes

Study 71

Routing Non-routing (traditional)

How Routing Works: Big Picture

SOURCE 1

SOURCE 2

SOURCE 6

SOURCE 5

SOURCE 4

SOURCE 3

SOURCE 7

Multiple Sample Sources*

Survey D

Survey E

Qualified Respondents Sent to Surveys

Router – Determines Survey Qualification

E

B

A C

D

Survey C

Survey B

Survey A Screening for Survey…

* Panel and/or river sourcing – using a variety of management techniques.

Changing Market / Internet Environment

Continued decline of response rates

Changing Internet landscape and respondent behaviors

Continued growth of online studies

Key challenge facing the online industry today:

Sample Capacity

Consequences of doing things the “the way we’ve always done it”

Inefficient use of sample – capacity not optimized Respondent engagement suffers – lowering capacity Projects are increasingly harder to complete

Why is the Industry Adopting Routing?

Benefits Noted by Providers

Increased Sample Capacity & Population

Reach

Easier Survey Adaptation to New

Online Realities

• Allows studies to be tailored to people accessing them using different devices (e.g., mobile)

• Allows more sophisticated management of a broad base of studies, and the balancing of sample supply and demand (e.g., applying usage rules to sources, and exclusions to respondents)

• More efficient use of sample in general, and especially lower-response demographics

• Allows access to parts of the population that don’t traditionally join panels

• Greater control over operational aspects (e.g., automation)

• Automation produces greater sampling replicability

• Allows for measurement and control of selection bias (inherent in all methodologies, including panels)

• Increased respondent engagement and satisfaction

Greater Consistency & Quality Control

Findings from Router and Sample Provider Focus Group

Some Definitions Before We Start…

• Screening

A set of questions usually asked/utilized prior to main questionnaire content to determine or verify study eligibility or branching within a specific study.

• Reallocation / Reassignment The act of reassigning a respondent to further screening upon disqualification during an

attempt.

• Priority Assignment The act of assigning a screener for attempted qualification in a non-random manner,

based on a set of predefined business rules.

A full Glossary of router-related terms was created by the Router team and is available now!

Serial vs. Parallel Routers

Routers in use today can be broadly classified as either Serial or Parallel

Insight / Conclusion

Routers have variables or characteristics that need to be managed according to business rules


Router Team Findings

Randomization & Prioritization

Common screening

Studies - # and type

Frequency of survey-taking

Study qualification correlation

Parallel or Serial

Reallocation

Such as…

The Common Survey Environment

There is selection bias in all methodologies, but in a router we can more easily measure it

Screening process is shared among studies

Respondents may qualify for multiple available studies

Studies “compete” with each other in terms of access to respondents

Some studies will need priority over others in getting respondents (e.g., if they only need a few more completes)

Selection bias comes from other studies taking priority over your study for respondents

Characteristics

Routers use Randomization and Prioritization for survey assignment, usually in combination


Router Team Findings

Based on business rules for a router

Used in varying degrees by different routers

• 100% randomization best minimizes bias, but may not be practical

• 100% prioritization usually leads to greatest bias

• Hybrid approach is most commonly used

Quick Review of Methodology

Created a router to test various scenarios in a controlled fashion

Four “live studies” and 20 screeners from other studies = 24 studies in router

The 4 live studies were chosen to represent diversity across the market research business

• Concept Test (CPG), Customer Satisfaction (Telecommunications), Usage & Awareness (Health), FOQ2 questionnaire (shortened to 15 minutes)

The 20 screeners were chosen to represent diversity typically seen in a large routing system

Respondents were randomly assigned to either the Router (Test) or Non-Router (Control) conditions

Mirrored the 4 live router studies with the same 4 studies fielded in the traditional way (non-routed)

Router condition: Respondents were assigned to either a Serial or Parallel screening process

Both direct comparisons and simulations were used to derive insights from the data

What is the impact of moving to a common survey environment?

Let’s look at some data!

Research Questions

•Compared bias in the Serial and Parallel conditions through simulations, for 16 key metrics

•Compared survey data responses of Test (Serial and Parallel) conditions to Control condition for: 6 demographics, 12 behavioral variables, 10 attitude questions / batteries, 6 key survey metrics

•Compared length of time spent in the screening process

Does routing impact the data?

•Compared satisfaction with the survey experience for Test and Control respondents

Does routing impact the respondents?

•Examined sample capacity lifts due to routing

Does routing help sample capacity?

Is There Selection Bias in Routing?

We saw very little bias overall in both Serial and Parallel routing


Some variables are more biased than others

But did the bias we saw matter – did it affect survey responses ?

Distribution of Error Across All Surveys (Worst Questions in Each)

Router No Router

Does Routing Impact the Data?

Routing did not produce meaningfully different results vs. the traditional method

Of 252 comparisons, 26 were significant (10%)

Within what we would expect by chance given the large # of comparisons we tested.

30

138

72

12 2

14 8

2

Demographics Behaviors Attitudes Key metrics

# of comparisons

# significant


Does Routing Impact the Data?

About the same amount of comparisons for Serial and Parallel were different from the Control

Sometimes Control was higher, sometimes it was lower

# of significant comparisons 26

Control vs. Parallel 12

Control higher 8

Control lower 4

Control vs. Serial 14

Control higher 3

Control lower 11

There was no consistent pattern of differences

Two Major Types of Screening Processes

Parallel Serial Screener 42

Screener 23

Screener 71

Screener 8

QUALIFIED:

Screener 71

Screener 8

NQ:

Screener 42

Screener 23

Selection of Study 71

Screener 42

Outcome: NQ

Screener 23

Outcome: NQ

Screener 71

Outcome:

QUALIFIED

Enters and

completes

Study 71

Respondents attempt to qualify for one study at a time. If they don’t qualify for one study, they move to the screener for the next study.

Respondents attempt to qualify for several studies at one time. There can be several levels of qualification, but each level is for multiple studies

Enters and

completes

Study 71

Selection Bias for Serial Screening

For Serial routing, randomization is important. Adding 50% randomization decreases the bias by 20% on average.


-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

Error with 100% Priority Serial

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

Error with 50% Priority Serial

Bias (distance from 0%)

Bias is very small compared to

sampling error

Selection Bias for Parallel Screening

For Parallel routing, randomization is important but it does not matter whether it is at the pre-screening or survey selection level


-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

Error with Random/Priority Parallel

-20%

-15%

-10%

-5%

0%

5%

10%

15%

20%

Error with Priority/Random Parallel

Pre-screening level is random Survey level is random

Bias

Does Type of Screening (Serial or Parallel) Impact Data?

Serial and Parallel screening may produce meaningfully different results for some variables

Of 126

comparisons, 15

were significant

(12%)

A bit more than what we

would expect by

chance, even given the

large # of comparisons

we tested.

Serial was higher/more

positive than Parallel for

all 15 differences

Nine of these differences

were on one variable –

Looking for health

information online


Time Spent in the Screening Process

Respondents spend longer in a router screening process than the traditional screening process


Numbers are specific to the router setup used for this study

3.87

1.20

Router Control

Average minutes spent in screening

process

Time Spent in the Screening Process

Respondents spend somewhat longer in a Serial screening process than a Parallel screening process


Numbers are specific to the router setup used for this study

4.16

3.58

Serial Parallel

Average minutes spent in screening process, by condition

BUT DOES IT REALLY MATTER?

Does it impact survey data?

Does it impact respondent satisfaction?

Does Length of the Screening Process Impact Data?

Length of the Serial screening process did not impact survey data – respondents who spent longer did not answer differently

2.72

4.97

6.23

1.2

0

2

4

6

Brief Medium Long Control

Cell

Min

utes

Router

Control

Serial

Mean Minutes In ScreeningExamining the Serial condition (where respondents spent the most time)… Three screening length conditions were tested and compared to each other and to the control.


Length of the Screening Process

• 1314 pairwise t-tests run, across 37 tables of demos and KPIs.

2.5% were flagged as significant (less than what is expected by chance alone).

These differences were spread out across the variables (no concentration on any single variable).

0

500

1000

Actual Expected

StatusT

ests

Result

Not Significant

Significant

Number of Significant Stat Tests

Length of the Screening Process

There does appear to be an effect where males are more likely to drop out (i.e., not end up in the final sample) with longer screeners.

This may point to a need to control gender through quotas.

47.2

44

36.4

45.6

0

10

20

30

40

Brief Medium Long Control

Cell

Pct

.Mal

e Router

Control

Serial

Percent of Valid Respondents Who are Male

Does Routing Impact the Respondents?

Respondents were as satisfied with the routing survey experience as were control respondents with the traditional experience.

Time spent in the Serial screening process did not negatively impact enjoyment.

Screening % agree

Control 75%

Serial 74%

Parallel 73%

“I enjoyed filling out this survey”

Time spent in

screening (Serial)

% agree

Brief 74%

Mid 75%

Long 77%

Interestingly,

respondents who spent

the longest time in the

serial screening

process liked it

significantly more than

the Brief respondents!


How Does Routing Impact Sample Capacity?

The router’s overall incidence (the average % of studies for which a respondent qualifies) is what drives sample efficiency


33%

67%

Eligible

Non-Eligible

Within our study

router, on average, a

respondent was

eligible to complete

about 8-9 of the 24

studies

Routing Increases Sample Capacity

Routing makes more efficient use of respondents than fielding each survey separately


With 12 studies in our router, we only need

40% of the respondents

we would need if we were filling them all separately

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12

Ind

ex v

ers

us

s

ep

ara

te s

tud

ies

Number of studies in router

Respondents required on average versus fielding separate studies

Note: This is a conservative estimate! The actual impact would likely be higher for a larger router, and with new surveys being added as others complete.

Conclusions –Takeaways

Top Takeaways

Selection bias exists, but it is small, less than sampling error, and varies by question

Routing does not impact survey data in any meaningful way

There may be differences on some variables between Serial and Parallel routers

Quotas may be needed to control response differences by demographics

Longer time spent in the screening process does not impact survey data

Respondents were happy with the routing process, even though it took slightly longer

Routing increases sample capacity by making more efficient use of respondents

1

2 3 4

5 6

7

Recommended Practices

Reduce cognitive load and irritation for

respondents

• Standardized wording for screeners

• Store respondent profile information

• Give them periodic opportunities to leave the screening process,

• Length of screening process needs to be reasonable

Consistent and frequent monitoring & management

• Router business rules are important

• Rules are established by sampling experts

• Controls are in place to ensure rules are followed

What You Should Do Differently Tomorrow

• Ask your suppliers if they are using routers on your studies

• If so, use the checklist created by the router team to better understand the details

And…

• We encourage you to always ask questions about your sample - better sample leads to higher quality data, which leads to better business decisions!

APPENDIX

Serial Routing

Definition In its most basic form, serial routing is a process whereby a respondent is screened sequentially for the available studies on a router. Upon qualification for a study, the respondent is immediately sent into the full survey. Upon disqualification in any screener, another screener is served to the respondent – with the process repeating until qualification occurs.

Example Respondent enters a router, and is randomly served the screener for Study 42. He does not qualify and attempts Study 23, resulting in another disqualification. The router then randomly serves the screener for Study 71, the respondent meets the criteria, and is sent into Study 71 for a full and joyfully robust survey experience.

Parallel Routing

Definition In its most basic form, parallel routing is a process whereby a respondent is exposed to a set of screening questions from all or a subset of the surveys on the router, simultaneously, on the same page. After the respondent answers these questions, he or she is assigned to one of the studies for which he or she appears to be qualified.

Example Respondent enters a router, and a page of questions is presented that contains the screeners for Study 42, Study 23, Study 71, and Study 8. The questions were selected at random by the system with presentation order randomized. After completion of the page qualification is tested, with the respondent qualifying for only Study 71 and Study 8. Study 71 is randomly assigned for a full survey experience.