Download - Game-Changing Audience Research Techniques
Game Changing User Research Research as cheap & easy as a cup of coffee
PRESENTED BY ALISSA RUEHL & NATHAN RENIHAN
TWEET!
#BBCON
#UserResearch
@ALISSARU
WHY?
People in the world
Visitors to your site
Visitors to your donation
form
Donors
People in the world
Visitors to your site
Visitors to your donation
form
Donors
People in the world
Visitors to your site
Visitors to your donation form
Donors
People in the world
10% more
Visitors to your site
10% more
Visitors to your donation form
10% more
Donors
People in the world
Visitors to your site
10% more
Visitors to your donation form
10% more
Donors
People in the world
Visitors to your site
Visitors to your donation
form
10% more
Donors
As an insider
your perspective is very different.
Bite sized
research
PROOF
• We’ve taken our own medicine
• Onsite research:
- Total time: 20 minutes
- Total cost: $0
vs
OUR USER RESEARCH
4 11
WHAT?
What’s the mystery you’re trying to solve?
• Why are people abandoning the donation form?
• Why don’t more people give recurring gifts?
• What are the main things people are looking for on
my site?
• Do people notice the promotion we’re doing?
• How should we organize our new website (or
newsletter, etc)
POTENTIAL CASES
WHO?
People involved with your organization
The general public
PARTICIPANTS
• Good targets
- People somewhat involved with your organization
• Event participants
• Donors
- The general public
• Avoid
- Employees
- Board
• Generally avoid*
- Major volunteers
- Friends & family
HOW?
•Starbucks testing
•Card sorts
•Surveys
3 QUICK & EASY METHODS
STARBUCKS TESTING
When you want to know if average people run into
obstacles when trying to complete a task
CARD SORTS
When you want to know how people think information
should be organized
SURVEYS
When you want to answer specific questions about
people currently involved with your organization
STARBUCKS TESTING
When you want to know if average people run into
obstacles when trying to complete a task
RULES FOR TESTING
TASK BASED
VERBAL FOLLOW UP
IT WORKS ABOUT LIKE THIS
ALWAYS BE PREPARED
1. Make sure you pick something that regular people
use, you will get regular people at Starbucks.
2. Test internally for practice
3. Test internally to make sure it is truly a 5 minute
test
4. Take 1 person with you. 1 person stays with laptop,
1 person solicits.
BEFORE YOU GO
1. Make sure Starbucks is ok with it. Talk to the
manager on duty first, have a pitch ready for that
conversation so you can easily explain what you
want to do.
2. Bring your charger, mouse, test internet connection
(if needed), etc.
3. Grab a table - go early or avoid rush times.
BEFORE YOU START
1. Talk to the people that are sitting around, you are buying
their NEXT cup of coffee, avoid the people in line.
2. Put your best foot forward, pick someone that will get a
good response.
3. Work on your pitch, it has to be faster than an elevator
pitch. Think of your pitch like a tweet: short, sweet and
to the point.
4. Make it obvious that you are a non-profit, lead with that,
wear the gear, beat people over the head with it; people
are more likely to help.
5. It helps to stress that this will take 5 minutes or less.
WHEN YOU ARE READY
1. Don’t record names
2. Do get a wide cross section of users (age,
gender, race, etc)
3. Test the savvy level of the user, make sure they
aren’t extremely savvy or extremely novice.
4. Give them their task
5. WATCH
6. Try not to help
7. Ask follow up
WHEN YOU HAVE A USER
• If you find a massive gotcha, STOP. No need to keep
testing. Go back and fix it and try again another day.
• Once you have 3-5 users you probably have some really
good feedback, no reason to keep going. Maybe switch
to a new thing to test.
• This whole process might take 1.5-4 hrs
• You will most likely learn things that will blow your mind.
Things you never would have thought of.
WHEN THE TEST IS OVER
QUESTIONS ON STARBUCKS TESTING
CARD SORTS
When you want to know how people think information
should be organized
People in the world
Visitors to your site
Visitors to your donation
form
Donors
07/10/2013 44
07/10/2013 45
07/10/2013 46
07/10/2013 47
vs
OUR USER RESEARCH
12 3
• Write down each 2nd level menu item on
its own note card
- In this example
• Write down
– News Archive
– Wayne’s blog
– Press releases
– Media contacts
• Don’t write
– NEWS
CARD SORTS
1. Get a participant and some table space
2. Give them
- your note cards
- some blank cards or post-it notes
• (No more than you have space in your top level
navigation)
3. Let them sort the pre-written note cards into
groups
4. Ask them to label each group (using the
blank note cards)
5. Record the results
6. Repeat at least 10 times
CARD SORTS
07/10/2013 51
• Tips
- Be sure to stay quiet while the participant is working
- Ask them to think aloud and listen to their commentary
- Take a photo when each person is done to capture their
work
• Variations
- Pre-write the headers (closed card sort)
- Conduct the card sort online
• http://www.optimalworkshop.com/optimalsort.htm
CARD SORTS
QUESTIONS ON CARD SORTS
SURVEYS
When you want to answer specific questions about
people currently involved with your organization
POLL: WHO HAS CREATED OR SENT A SURVEY?
POLL: WHO HAS ABANDONED A SURVEY THEY’VE
RECEIVED?
• 11 out of 15 people surveyed had never done any
user testing
- Surveys were the way most people do user
research
• 15 of 15 people surveyed had abandoned a survey
half way through because it was too long or too
confusing
OUR USER RESEARCH
• Keep it short
- Usually have 5 – 15 questions
- Should take less than 5 minutes to complete
• Always have a general comment field
- Let them speak to you, not just answer questions
• Have someone proofread your survey
- Make sure none of the questions are confusing
• Be neutral
- You want information, not vanity metrics
• You don’t need to survey everyone
- How many responses do you need to answer your questions?
• You can use statistics, or
• General questions, homogeneous audience ~100 responses
- Assume a response rate of 10 – 20% for people somewhat involved with
your organization
SURVEY ETIQUETTE
• Think of each question as testing a hypothesis. It
can help to word them that way
- Less useful: We should ask how many times per year they visit our
museum
- Better: I suspect that most people visit either only once per year or
more than 4 times. This will affect how we price and market our
membership programs. If we find a lot of people who visit 3-4 times
a year, we might decide to adjust our programs
SURVEY ANALYSIS TIPS
• Avoid “neutral” options
- Very likely
- Somewhat likely
- Somewhat unlikely
- Very unlikely
• (But you can use “not applicable”, if applicable)
• Experiment with forced rankings
• If you’re doing a large survey, consider how many open-ended
questions you want to analyze
SURVEY ANALYSIS TIPS
QUESTIONS ON SURVEYS
OTHER THINGS
WEB ANALYTICS
When you want to know how people are getting to your
site & what they are doing on your site
07/10/2013 71
07/10/2013 72
Focus Groups
07/10/2013 73
Formal Usability Studies
07/10/2013 74
Blackbaud Discovery
THIS PRESENTATION BROUGHT TO YOU BY
BLACKBAUD USER EXPERIENCE
• We want to learn from you so we can design
experiences that delight
• Participate in Blackbaud Discovery!
- Help us with our research
- Have your voice heard for the products you use
• Sign up to participate at
www.blackbaud.com/discovery
• Stop by the Discovery booth in the expo hall
FOR MORE INFORMATION
LINKS
• Alissa’s blog posts on npengage
- http://www.npengage.com/author/alissa/
- http://www.npengage.com/online-fundraising/testing-your-donation-
form-part-1-why/
- http://www.npengage.com/online-fundraising/testing-your-donation-
form-part-2-designing/
• Other posts
- http://www.npengage.com/web-design/nonprofit-web-design-process-part-
surveys-and-interviews/
LINKS
• Analytics
- http://www.google.com/analytics/
- http://www.npengage.com/content-marketing/google-analytics-1/
• Starbucks testing
- http://www.mattbaline.net/3-lessons-from-guerrilla-mobile-usability-at-
starbucks/
- http://joshledgard.com/i-wish-id-thought-of-that-operation-starbucks/
- http://blog.kooaba.ch/2012/02/developing-shortcut-starbucks-usability-
testing/
• Card sorting
- http://rosenfeldmedia.com/books/card-sorting/
– http://www.optimalworkshop.com/optimalsort.htm
• General
- http://uxdesign.smashingmagazine.com/2009/05/15/optimizing-conversion-
rates-its-all-about-usability/
- http://www.nngroup.com/articles/non-profit-websites-donations/
- http://www.sensible.com/rsme.html