recruiting better user research participants: upa 2011

© 2010. ELECTRONIC INK, INC. ALL RIGHTS RESERVED. 1Recruiting Better User Research Participants

Presented by Jim Ross

23 June 2011

Recruiting Better User Research Participants

Principal of Research & StrategyElectronic Ink

UPA 2011

23 June, 2011

© 2010. ELECTRONIC INK, INC. ALL RIGHTS RESERVED. Recruiting Better User Research Participants 2

Why talk about recruiting?

23 June 2011

User research is only as good as the participants you recruit.

It’s difficult to recruit participants who …•Are representative•Can express themselves and provide useful information•Show up



23 June 2011

Have you ever …•Recruited someone who didn’t fit the profile or was just odd?



23 June 2011

Have you ever …•Recruited someone who lied to get into the study and was just in it for the money?



23 June 2011

Have you ever …•Recruited people who couldn’t express themselves?



23 June 2011

Have you ever …•Had too many no shows and cancellations?



23 June 2011

What other recruiting problems have you had?


What we’ll talk about

23 June 2011

Recruiting for moderated research•Who should do the recruiting? •Writing a better screener•Minimizing no-shows•Recruiting backup participants

Recruiting for unmoderated, remote research•Emailing participants•Using an online panel•Using online intercepts


Recruiting for Moderated Research.

23 June 2011



23 June 2011

Who should do the recruiting?


Who should do the recruiting?

23 June 2011

Recruiting options• You• Your client• A recruiting company



23 June 2011

Doing the recruiting yourself



23 June 2011

Advantages• You know what you’re looking for.• You have more control over who is recruited.• You sound more human and less like a telemarketer.• You save money.



23 June 2011

Disadvantages• It’s difficult if you don’t already have a list to work from.• It’s time consuming to find, contact, screen, and schedule.• It may require a lot of cold calling.



23 June 2011

Start with lists of potential participants - customer lists, membership lists, employee directories, etc.


Or gather participants from your website with a recruiting page or an online intercept.


23 June 2011


Or advertise or connect through social media and other sources.


23 June 2011



23 June 2011

Having your client do the recruiting



23 June 2011

Advantages• Clients have access to employee, customer, and member lists.• Employees, customers, or members are more likely to volunteer.• It saves you money and time.



23 June 2011

Disadvantages• Clients may not be very good at recruiting.• Clients may not possess the time and organizational skills required.• Clients often pass on vague or incorrect information about the research.



23 June 2011

Provide your client with detailed instructions.• Profile of who to recruit• And who not to recruit



23 June 2011

Provide your client with detailed instructions.• Schedule of sessions



23 June 2011

Provide your client with detailed instructions.• Description of the study to give to participants



23 June 2011

Ensure they do it correctly.• Delegate to someone with time and organizational skills. • Monitor the recruited participants. • Email the scheduled participants to introduce yourself and provide more

details.



23 June 2011

Using a recruiting company



23 June 2011

Advantages• It saves you a lot of time and effort.• It’s a good option when you have no existing lists or other resources.• With the right information, a good recruiting company can get the best

participants.



23 June 2011

Disadvantages• It’s the most expensive recruiting option.• You might get “professional” participants who are only in it for the money.• With poor direction or a bad recruiting company, you can get the worst

participants.

http://www.savitzpanel.com/sp1.asp

http://www.savitzpanel.com/sp1.asp



23 June 2011

Selecting a recruiting company• Get referrals from other researchers.• Get quotes from several recruiting companies - cost and confidence.• Look for good communication.• Cultivate several recruiting companies you can use regularly.



23 June 2011

A strong and clearly written screener is essential.



23 June 2011

Review the list of recruited participants.• Ensure they fit what you’re looking for.• Have your client review and approve the list also - CYA!



23 June 2011

Deciding who will do the recruiting



23 June 2011

Do your own recruiting when:• You have more time than money.• You don’t mind or are effective at cold calling.• You can hire someone else to do the recruiting – intern, staff recruiter.• You can use existing lists, recruit from your website, or advertise.• You already have a connection with potential participants.



23 June 2011

Have your client do the recruiting when:• Your client has detailed contact lists of employees, customers, or

members.• Your client already has a connection with potential participants.• Your client has the time and coordination to do this effectively.• Your client has successfully done recruiting in the past.



23 June 2011

Hire a recruiting company when:• You have more money than time.• You don’t want to “cold call” and screen participants.• You don’t have an easy way to find potential participants.



23 June 2011

Writing a better screener



23 June 2011

Keep people from hanging up on you• Quickly get to the point.• You’re not selling anything.• You’re recruiting for a paid study.



23 June 2011

Ask likely elimination questions first• Only the most promising will go through all the questions.


Eliminate the usual suspects• Eliminate recent participants.

How would you answer this question if you wanted to get into the study?


23 June 2011


Eliminate the usual suspects• “Professional” participants might lie.• Recruiters should check their records before calling people.

“A few years ago.”


23 June 2011



23 June 2011

Eliminate conflicts of interest• Existing relationship with your client - employee, employee of a

competitor, etc.• Too much inside knowledge - web designer, developer, marketer, etc.



23 June 2011

Avoid reading multiple choice answers• The “correct” answer is often obvious



Avoid reading multiple choice answers• The “correct” answer is often obvious

“None of these”


23 June 2011


Ask open-ended questions instead of reading the answers



23 June 2011


Ask open-ended questions instead of reading the answers• Requires the recruiter to know which professions should terminate the

screening


23 June 2011



23 June 2011

Don’t make the elimination answers obvious



Don’t make the elimination answers obvious

“19 – 24” seems like a safe answer.


23 June 2011


Include multiple elimination answers.



23 June 2011


Include multiple elimination answers• It’s not obvious which answers will get you eliminated.


23 June 2011



23 June 2011

Avoid giving away what you’re looking for




23 June 2011

Avoid giving away what you’re looking for

“Yes”


Avoid giving away what you’re looking for• Providing many choices obscures what you’re looking for



23 June 2011


Avoid giving away what you’re looking for• Providing many choices obscures what you’re looking for


23 June 2011


Avoid giving away what you’re looking for• Beware of those who choose everything to try to get into the study.


23 June 2011


Avoid giving away what you’re looking for• Set a reasonable limit to eliminate fakers.


23 June 2011


Avoid giving away what you’re looking for• Use open-ended questions to completely hide what you’re looking for.


23 June 2011



23 June 2011

Ask questions that only qualified participants can answer



23 June 2011

Require proof• Require participants to provide proof of qualification.• Provide a graceful exit, so that people don’t have to be caught in a lie.



23 June 2011

Recruit based on behavior and attitudes• Most demographics aren’t very important for small-scale research.



23 June 2011

Recruit based on behavior and attitudes• Screen for desired behaviors.



23 June 2011

Recruit based on behavior and attitudes• Negative attitudes towards your client or subject matter may disqualify

people.



23 June 2011

Recruit based on behavior and attitudes• Screen for desired attitudes.



23 June 2011

Recruit based on behavior and attitudes• Interpret attitudes and behaviors based on answers.

What do you think we’re looking for here?


Recruit based on behavior and attitudes• Interpret attitudes and behaviors based on answers.

People interested in Japanese cars


23 June 2011



23 June 2011

Don’t ask questions that are difficult to answer

Can you answer this question easily and accurately?


Don’t ask questions that are difficult to answer• Estimating the average number of hours per day is easier.

Is this question easier to answer?


23 June 2011



23 June 2011

Don’t use the screener to gather information• Only include questions that screen participants.• Save information gathering for a pre-study questionnaire.



23 June 2011

Screen for computer, Web, and technology experience• You may want to eliminate those with very low and very high

experience.


Screen for computer, Web, and technology experience• Self-assessments allow those who are truly uncomfortable with

technology to excuse themselves from the study.• But don’t rely only on self-assessments.


23 June 2011



23 June 2011

Screen for computer, Web, and technology experience• Make your own assessment based on activities and behaviors.



23 June 2011

Screen for eloquence• Ask an open-ended question related to your topic to judge

expressiveness.



23 June 2011

Screen for eloquence• But be careful not to eliminate qualified, quiet participants.• “Normal” people may not be as entertaining as “professional”

participants.• Recruit representative participants, not only those who speak in perfect

sound bites.



23 June 2011

Ensure that people are physically able to participate



23 June 2011

Let people know what they’ll be doing• Eliminate surprises and potential discomfort.• Allow those who would be uncomfortable to back out.



23 June 2011

Test your screener with a colleague• Ask a colleague to try to fake his/her way through the screener.• Recruiting company should point out problems in logic.



23 June 2011

Get your client to approve the screener – CYA!• Your client becomes responsible for the types of people recruited.• It’s difficult for them to complain later about the quality of the

participants.



23 June 2011

Minimize No-Shows


Minimize No-Shows

23 June 2011

Locate your research conveniently.• Where are your participants usually located during your session times?• How easy is it to get to and find your location? • Traffic? Public transportation? Parking?• Is it more convenient to go to your participants or conduct remote

sessions?

Jeanne Adamson

Nice place for an Electronic Ink plug :)


Minimize No-Shows

23 June 2011

Schedule your research conveniently• What are your ideal participants usually doing during your session

times?• Schedule a variety of session times.• Who’s most available for weekday research?


Minimize No-Shows

23 June 2011

Avoid days and conditions that result in more no-shows• Fridays• Days before holidays• Bad weather – snowstorms, rainstorms, etc.


Minimize No-Shows

23 June 2011

Provide appropriate incentives.• Make it worth their while to show up.• Make it appropriate to the person and to the activity.


Minimize No-Shows

23 June 2011

Contact participants to confirm• Remind them to attend.• Give directions and parking / transportation information.• A good way to receive early warning of no-shows and get a head start

on recruiting replacements



23 June 2011

Recruit Backup Participants



23 June 2011

Two approaches to backup participants• Floaters• Over-recruiting

Floaters Over-recruiting



23 June 2011

Floaters• Wait on standby to fill in for no-shows or inappropriate participants



23 June 2011




23 June 2011

Over-recruiting• Recruit several extra participants to use for no-shows• Dismiss and pay those you don’t need



23 June 2011




23 June 2011

Use floaters when• You want to be sure your observers always have a participant to

observe• You don’t have time for an extra day of backup sessions• You don’t have money to pay for an extra day of sessions at a research

facility



23 June 2011

Use over-recruiting when• It’s not important that your observers have a participant to watch for

each session• You have time and money for an extra day of backup sessions• You want to save money on incentives• Your participants don’t want to wait around as floaters


Recruiting for Unmoderated, Remote Research.

23 June 2011


Recruiting for Unmoderated, Remote Research

23 June 2011

What is different about unmoderated, remote research?• Large numbers of participants from anywhere in the world can

participate• Participants visit a website to complete the study on their own time• Unmoderated tools automatically capture and present the data



23 June 2011

With a large number of potential participants, different methods are required to: • Find• Screen• Contact• And convince people to participate



23 June 2011

Three recruiting methods for unmoderated studies

Emailing participants

Online panel Online intercept



23 June 2011

Emailing Potential Participants



23 June 2011



23 June 2011

Gather a list of potential participants• Use existing lists of customers, members, employees, etc.• Recruit from your website• Advertise



23 June 2011

Narrow the list of participants using existing information.



23 June 2011

Avoid adding screening questions to your study• Each question adds to the length of the

study, increasing abandonment.



23 June 2011

Determine how many people to email• Determine the minimum number of participants for acceptable results.

- Tullis and Wood showed 30 - 50 is needed for online card sorting. [1]

• Assume a low response rate of 5% and email an appropriately larger population.

[1] Tullis, Tom, and Larry Wood. “How Many Users Are Enough for a Card-Sorting Study?” Proceedings UPA 2004, Minneapolis, MN, June 7–11, 2004. Retrieved March 7, 2011: http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf

http://home.comcast.net/~tomtullis/publications/UPA2004CardSorting.pdf



23 June 2011

Determine how many people to email• Determine the maximum number of participants to accept.• Avoid paying too much in incentives.



23 June 2011

Keep it short, briefly describe the study, and provide a link to the study



23 June 2011

Don’t provide the study instructions in the email



23 June 2011

Highlight the incentives, both monetary and altruistic



23 June 2011

Provide appropriate incentives: individual incentives are small but add up.



23 June 2011

Provide appropriate incentives: entry into a drawing is often easier.



23 June 2011

Provide appropriate incentives: with equal appeal to everyone.



23 June 2011

Establish credibility with individual emails in a friendly, professional tone.



23 June 2011

Establish credibility by providing contact information.



23 June 2011

Establish credibility by sending the emails from a familiar person.



23 June 2011

Monitor participation and send reminders• Monitor the number and type of people who have completed the study.• If necessary, send reminder emails.



23 June 2011

Using an Online Panel



23 June 2011

A panel is a database of people who have volunteered for surveys.



23 June 2011

Panels are provided by many recruiting and market research companies.



23 June 2011

Panel members are already pre-screened• You provide the details of the people you’re looking for instead of a

screener.• The recruiter filters the database to find people matching those

characteristics.



23 June 2011

Screening• You can also add additional screening questions to the beginning of

your study.• Panel participants are motivated to take surveys, so you don’t have to

worry as much about extra questions leading to abandonment.



23 June 2011

You pay the panel company based on the number of “completes.”• For each person who completes the study, you pay a recruiting fee and

small incentive.• Limit your costs by ending the study after a maximum number of

participants.



23 June 2011

Quality of panel participants• They want to participate in studies, and they want to be invited back.• This motivates them to take the study seriously and provide thorough

feedback.



23 June 2011

Quality of panel participants• They are more likely to be “professional” participants, primarily in it for

the money.• They may not represent your typical user.



23 June 2011

Using Online Intercepts



23 June 2011

Online intercepts pop-up on websites to invite people to participate.



23 June 2011

Those who accept are taken to a brief screening questionnaire.



23 June 2011

Those who pass screening can either be …•Added to a participant database •Contacted immediately for a remote, moderated study•Taken directly into an unmoderated study



23 June 2011

Creating an online intercept•Some unmoderated tools provide the ability to create intercepts.•Otherwise you may need a developer to create the intercept and tie it into the site.



23 June 2011

Advantages of online intercepts• They recruit “real users” who are already on the website.• Automatic screening and recruiting saves time.• They can remain on a website indefinitely, continually recruiting for

long-term studies.



23 June 2011

Disadvantages of online intercepts• Most people ignore them and don’t volunteer.• Intercepted participants have less motivation to complete the study. • Require a lot of site traffic or waiting a long time to get enough

participants.



23 June 2011

Incentives• Providing incentives to each participant is difficult logistically and

financially.• Using prize drawings may be an easier, less expensive incentive.*

* Be careful of the legal issues surrounding contests and sweepstakes. Consult a lawyer if needed.



23 June 2011

Choosing Between These Methods



23 June 2011

Email the participants if …• You have access to a list of potential participants.• You don’t need to do much additional screening.• The emails can be sent from someone the participants know.• You have more time than money.



23 June 2011

Use an online panel if …• You can’t easily get a list of potential participants. • You need a very specific type of participant, which would require a lot of

screening.• Your study requires a lot of time and effort for the participants to

complete.• You have more money than time.



23 June 2011

Use online intercepts if …• You can’t easily get a list of potential participants.• You want to use your site’s current users.• Your site gets enough traffic or you can wait for enough responses to

accumulate.• You’ll be running ongoing studies on your site.


Recruiting Better User Research Participants

23 June 2011

Conclusion


Conclusion

23 June 2011

Recruiting can be difficult, but it’s a crucial first step for effective research.• It’s worth taking the extra time to ensure that you get representative

participants, who provide useful feedback, and who show up!


Questions and Discussion.

23 June 2011

recruiting better user research participants: upa 2011

Business