Report on Focus Group Findings

ARF FOQ 2 Router Initiative

ABSTRACT

This report will detail the findings of a focus group conducted on behalf of the Advertising Research

Foundation on August 8, 2012 in New York City. The discussion was led by Steve Gittelman.

Those taking part in the discussion were chosen from a group of online sample users including account

managers and researchers drawn from a random sample of the top 50 Honomichl companies. Eight

companies participated in the forum. All participants met the following criteria:

1. Have direct client contact

2. Be responsible for proposing and or designing research projects that involve online sample

3. Be accountable to clients for online research projects conducted by their firms

4. Have some say in the design and online sample employed for projects

5. Have some familiarity with routers

While participants were not required to have employed routers for studies for which they were

accountable, or to have ever used routed sample, they had to be familiar with them. Also, some

participants were familiar with routers, but had rejected their use.

The subject of interest was participants’ thoughts and comments on the use of sample routers, a

technology-based approach to assigning sample to surveys. Routers typically:

1. Maximize the likelihood that anyone who wants to do a survey has the chance to do so

2. Increase the chances of filling all quotas and delivering within project schedules

3. Automate sample frame design to incorporate a set of well thought-out rules so that the

process is less ad hoc than in the past (when this was done manually, often by project managers

and in inconsistent ways)

4. Centralize decision making about how to optimize use of the available pool of respondents and

provide metrics so that decision makers are aware of how the router is performing.

Participants recognized that routers could increase the number of interviews that a sample provider can

yield, thereby lowering costs. They also understood that routers could provide a better experience for

respondents since there are fewer screen outs and a higher degree of participation in surveys – both of

which can lower respondent frustration. However, participants expressed apprehension regarding the

use of routers as the bias that routers can introduce into the sample frame remains unclear.

Although it was accepted as fact by the participants that there were benefits to be gained by the use of

routers, the researchers were alarmed by the lack of transparency from sample or technology providers

regarding their use. Many have surrendered to the concept of having sample providers manage sample

frames despite the lack of training that the technologists and sample provider project managers receive

in this area. However, when asked what diagnostic information they believed would be useful for them

in managing and overseeing the process, they expressed a desire for information but also sense of

frustration in that even if they received the information that they were asking for, they were not

confident that they knew how to use it to manage the process and alleviate any potential bias in the

sample frame.

Prepared by: FOQ 2 Router Initiative Team, including primary authors: Steve Gittelman, MKTG Inc. Efrain Ribeiro, Lightspeed Research Direct quotes from focus group participants have been reflected as well as audio recording allowed.

Discussion

Sample Sources

One of the first topics discussed was where participant companies obtained their sample. The general

consensus was that they obtained their sample from a multitude of sources and they were willing to cite

some of the names of their providers (sources). Some chose to single-source studies, allowing the

sample provider to use partners to fulfill quotas when needed. This was most evident with researchers

who were employed by large companies possessing their own internal sample sources (which may be in

a different entity within the larger company). One participant offered: “Supplier X currently handles

about 60 something percent of our volume. And then they decide on partners.”

The main domestic players cited included Lightspeed, SSI, Research Now, Federated, uSamp, GMI,

Borderless Access and M3. For international research, Research Now was cited as the principal player

mentioned.

Consistency was most frequently mentioned as a reason for selecting a sample source. “That’s a big

concern of ours, when we do our work, is [to] make sure that we have consistency in the data and what

we’re actually sourcing [… ].“

Another reason given for choosing a sample provider is control of the process. As one participant stated

“We don’t drop off our sample order and pick up our completes the next day. Our selection process is

based on partnerships that allow us to drive [...] panels in the way that we need to deliver a particular

data set to a client.”

Feasibility

Feasibility was also cited as an important factor to consider when choosing sample providers. One

participant indicated she would be “Very unhappy if my supplier can’t tell me upfront that they’re not

going to be able to achieve it [quota].” Although she admitted that she had limited experience with

routers, she believed that routers could help with feasibility.

For one participant, panel selection is based primarily on feasibility. That participant indicated that they

would rather go with a panel supplier who can complete the entire study then to have to split it up.

“Our approach is simply based on feasibility. If you know, based on the target audience […] if one panel

can do it for the life of the study then they do it simply because we haven’t seen data one way or the

other that proves or disproves that more sources [are better]. And, then based on the point of

consistency [...] is extremely important to us”.

Another participant mentioned that at the end of the day, when she needs to find the last few

respondents to finish the study, she will turn a blind eye and tell the sample provider to just finish the

job and find them. “[…] hide my eyes, close my mouth -- get me those 50 people, and I don’t care how

you do it.”

One participant expressed that router usage aids in feasibility, “[A router] undoubtedly helps with

feasibility”, but was concerned that routers can “[…] actually knock my measurements out.” Another

participant pointed out “So there are two things that routers do. One is to allow you to go to multiple

samples, multiple sources, so that’s definitely going to increase your feasibility. But also increase the

capacity of the internal panels as well because you’re optimizing people to go into certain surveys. So

everybody’s going to get a survey rather than not […].”

Types of sample used

Those companies that are not single-sourcing their online sample tend to seek diversity in the sources

they use. Similarly, those that are single-sourcing hope that the providers they use are drawing from a

diversity of sources. Some preference was expressed indicating that no single sample source dominates

the pool of sources used to fulfill the sample needs of any individual study. One concern of those

blending sources was that the same respondents do not repeat surveys. One participant stated, “When

we’re looking at sources, we’re looking at how often we have seen those people, how often they have

come in [to our survey]. And if we can apply that methodology on top of a router [...] control

[duplicates] no matter where they come from [...] you can actually control the quality you get.”

The overall sentiment of the participants was that with greater diversity came an increased likelihood

that a sample would be representative. “To me, I think you want to have as broad of a pipeline […]

beginning to end.” Companies want to use panels, but want the sourcing of the panel to be highly

diverse. They asked, “[if they} are coming on to the panel through River at first? If they are, that first

survey they answer will be different from when they get the next survey [...]. The key is to know how to

control the sources and consistently be able to put these measures into the study.” The participants all

seemed to agree with one participant’s statement that they “Think it’s much more difficult now to keep

representation than 10 years ago when everything was through email”.

Using social media as a source raised a security concern for one participant. She expressed an inability

to “train” respondents to avoid discussing the survey they had just taken with others. She feared the

possibility that Facebook participants could spread the word about the content of her surveys.

Routers

As indicated above, all participants had some degree of knowledge about of routers, while not all were

using them. All participants were in agreement that the use of routers can increase feasibility, but the

question remains “at what cost?”

Some stated that they currently use routers in their research and that they use the routers to control

the percentage of sample provided by the different sample sources and types. Those decisions along

with blending decisions are based upon the client and study. There are no hard and fast rules for these

blends; they seem to be determined on either a project by project, or company-wide basis.

One participant expressed that “Routers can be a respondent engagement tool”. It enables a

respondent who wants to take a survey to actually be able to do so. Some of the other benefits

mentioned included efficiency, cost savings and increased feasibility.

While all seemed to agree that routers had some definite benefits, there was also a great deal of

trepidation regarding their use. Trust was an issue brought up by a few of the participants. One

participant expressed that he doesn’t have trust in some of the routing providers to do it “the right

way.” Another explained that the problem she has with the routing providers is that they are not

researchers and their main priority is profit, stating “What they’re doing there -- their bottom line is to

make money based on selling the completes.”

There is also a good deal of concern with the number and types of studies that are being fed sample by a

router simultaneously. One participant was concerned by the impact that business and management

decisions of the sample provider might have on the prioritization of the studies within the router. She

felt that the router’s randomization technology worked, but if a particular study needed to be finished

and was then given a higher priority, data shifts could occur. She believed that the longer you leave a

project in field, the more random the resulting survey data would be. One participant was concerned by

the sizes of the various studies running in the router at one time and how that might impact the

“random” flow of sample. Another thought that a router would cease to be random if the percentages

of the different sample sources going through the router are changed so that one source contributed a

disproportionately large percentage of the sample flowing into the router process. She also said that

she “[…] did not realize I was buying a panel’s excess,” and concluded, “That’s horrible because it is no

longer random.”

Many of the participants expressed concern about the prescreening process and how it might impact

the survey data. The prevailing sentiment among participants was that between 5 and 6 was the

appropriate number of prescreening questions, although one participant said that “just a couple” was

appropriate. Others indicated they weren’t aware of the number of questions respondents were being

asked, but thought it was less than 10 with one stating “I mean, it might be 5-10. I can’t imagine they’re

much more than that.” There was uneasiness expressed with how many screeners a respondent would

see before they got to their survey.

Another participant wanted to know how many [full] questionnaires each respondent completed before

reaching her study. She thought she could add more screening criteria to adjust the routing assignment

and get more control. One participant looked at prescreening in an entirely different way and felt it was

good that the respondents could still take a survey if they were disqualified from a previous study. She

thought this would help avoid respondent frustration and provide a better experience for the

respondent. Yet participants were still “[…] concerned about [prescreening] taking us away from

randomization.”

Participants settled on a symbolic example that represented potential problems in a router. That

example was having a smoking and a toothpaste study running concurrently. All things being equal, one

can expect that essentially all of the population use toothpaste while a much smaller percentage of the

population smokes. An example was cited where the smoking study needed to finish and was prioritized

as the higher of the two studies. Once the respondent was pushed through the router for the smoking

study, assuming they qualified, they were removed from the sample pool for toothpaste. In a sense, the

toothpaste study could be completed by a lower number of people who smoked.

While most participants conveyed at least some concern over the hazards of using a router, none of the

participants had actually received any metrics from the sample providers that could show whether or

not their fears were justified.

Consistency

The number one concern expressed by participants was the maintenance of sample consistency. The

inability to produce a truly probabilistic sample is an inherent problem with all online sampling. The lack

of the ability to produce a probabilistic sample has left the researcher looking for some way to ensure

sample quality. The researchers in this group have turned to consistency.

They spoke about the consistency of survey data, the consistency in the blending of sample sources, and

the consistency of the sample sources themselves. They also discussed some of the methods they used

to maintain consistency. They admitted that they were not doing enough, but that having some control,

even if it is a limited, was better than no control. “That’s a big concern of ours when we do our work, is

to make sure we have that consistency in the data and what we’re actually sourcing.”

Several methods for sustaining consistency were cited. One strategy appears to be testing the

consistency of the sample coming out of the router. This method used both outside and internal

reference points, as well as consistency measures within the questionnaire as benchmarks. Brand usage

in particular seemed to be a prevalent benchmark.

A number of the participants preferred sample blends. One blend uses between 10%-15% river sample

and the remaining 85%-90% opt-in panel. These participants felt that maintaining a consistent blend of

sample sources would give them consistent survey data. One panelist cited “90% panel, 10% river […]

that’s my percentage […] as long as I know I can keep it that way, I know my data is staying consistent.”

Another thought the more sources used, the less chance there was for a sample to be biased. They

stated “[…] it is the law of averages protecting you a little bit. The more sources you have coming in, the

better off you are. If you can, you know, and using the router example again, if you could literally end

up with a million sources, then you’re better off.”

Another method discussed was the concept of pre-testing the panels before using them. While a

participant mentioned pretesting and comparing resulting survey data to sales data. He stated “The

other aspect on confidence that we have is we do validity in pretesting and in on our ongoing tracking

[...] at the end of the day they’ll take our pretesting results and they’ll compare it to our, you know, to

[...] sales data.”

One participant cites doing an analysis of his data by dividing it between different suppliers. If an

inconsistency is found, he would bring it up to the provider. He further expressed that although these

types of discussions with clients can be “difficult” on a project level, clients tended to appreciate these

types of conversations on an overall relationship level. If pretesting looks good, then they feel more

comfortable using a router [internal router] so there is more “hands on” management of the different

sources going through it.

Others are using clustering by asking the respondents both behavioral and demographic questions

during the survey. “The important thing is once you have defined those different clusters, the

consistency then from that point forward on every single study [...] what it comes down to is even on

trackers which is what Research Company X is really based off of – consistency is #1.”

Another participant described his company’s use of parallel tests -- essentially testing against a similar

study that did not use a router. “Well, in a sense, we just let them use the router to the full extent and

compared it to what we did previously when we didn’t use a router.”

While all the participants seemed to have a plan on how they are dealing with the use of routers, at

least some were willing to admit that what they were doing was probably not enough. They said that

their methods were not foolproof, but that they would rather do something than to do nothing. One

stated “So it kind of gives you a false sense of control, which I still prefer over no control.”

They were looking for some way to confirm that their methods of sustaining consistency actually

worked. They expressed their view that the needed information was either unavailable or conflicting. “I

think that’s one of our issues is that is the level of education that’s being provided by the owners of

these particular products. […] No one can come up with a valid answer to [help] alleviate some of those

concerns. We obviously have lots of people [...] that are all providing wide varieties of answers and you

know, [to] these particular questions.”

Others preferred to use a single panel as a way to maintain consistency, “I know what I’m getting. [I’m]

getting a sense of being able to replicate it.”

Metrics:

When asked which metrics they wanted to be made available to help them manage the routers and

sample sources, the participants had much to say. The key point was that the participants want to know

more about how the router technology itself works, including the sample sources being fed into them.

The problem is that they don’t know exactly what questions to ask.

Without knowing what else was happening within the router (other studies, volume, etc.) there was a

sentiment that they were unable to tackle the task themselves. One participant suggested that this was

the sample provider’s responsibility, while others thought that the panel companies were not

researchers and wouldn’t know if there was a bias being created in a survey by using a router. “They

don’t consider themselves necessarily research experts.” Another participant suggested that the sample

providers she worked with would like to do better, but needed help.

However, all could come up with at least a few metrics they wanted to know. One participant suggested

that she wanted to know “all of it,” and indicated that then she might be able to figure out what she

really needed to know, or perhaps there was so much going on that it doesn’t make any difference --

“Beats the hell out of me [...] yeah, I want to know everything. I also want to know why [...] it’s

important. Is all these different things [...] type of studies [...] does it make a difference? Maybe there’s

so much going on all the time that it doesn’t make a difference. I don’t know.”

A concern expressed was randomization vs. prioritization in the router and how the distinction impacts

the sample coming through to their study. Others wanted to know what the rules are, how many

screeners were used, how many surveys can a respondent take, prioritization, and the method for

invitation of or entry into the router. Another asked how and why the sample providers came up with

their rules, and seemed to be implying that their rules may be what works best for the sample provider’s

business model but not necessarily for the collection of quality survey data. They stated this is “… a

problem area. I don’t think you can trust the technology [provider] to come out and say here’s quality

attainable metrics that […] might influence your sample.”

Another thought “that it is too much information, and only the sample provider can manage the sample

itself, as you will never be able to know all the studies going through the router, how they impact yours,

how many are going through etc. […] is a tricky thing because you still have even with all of that

information […] there are still things that you just aren’t going to see.”

Participants want to know who their respondents are, and who are completing surveys and who are not.

Is their study going to be representative? Where is sample being sourced from? One participant

wanted some sort of certification, but wasn’t sure if that could actually be done. “I think it would be

nice if it eventually got to a point where it was standard so you could be accountable… Certification of

some sort would be even better, but I don’t know if you can get there.”

Summary

In general, the group was undecided regarding router use, and did not know if the benefits outweighed

the costs. While routers may provide benefits to respondents and projects as a whole, a better

understanding of their application and methods is needed to assess any impacts from their usage.

The group felt a responsibility to their clients to try and ensure the quality of the survey data they were

collecting, and better understanding routers would help achieve that. However, they felt compelled to

leave it up to the sample provider when it came to online sampling and the use of routers due to a lack

of understanding around sourcing and practices. The absence of knowledge complicates the situation,

and gives them even less control over the sample frame. In many cases the group didn’t even know

which questions to ask to gain such knowledge. While most seemed resigned to the fact that routers

are a technology that they are already using (and will likely need to use more in the future), some still

preferred to use a “pure panel” because they felt that they could better achieve consistency from a

known panel and its apparently less complex practices.

In the end, it was agreed that more knowledge and transparency about how routers worked was needed

by the industry. This includes an understanding of the methods used by routers, and how those

methods may or may not affect the data. Without this information, it is difficult for researchers to

manage their projects to ensure the quality of their data provided to clients.