webinar lab + ous presentation may10
DESCRIPTION
How does one pair online usability testing with lab usability testing? On May 6th, 2010, UserZoom joined forces with Bentley University’s Design and Usability Center to present how online, unmoderated usability testing can be combined with lab usability testing to produce rich, actionable results. We examined Amazon’s e-commerce grocery store and compared the results of each method to reveal the secrets and lessons we have learned.TRANSCRIPT
Webinar:
Combining Lab and Online Usability Testing: Lessons Learned
May 2010
Dial: 312-878-0211
Access Code: 599-198-841
Audio PIN: Shown after joining the Webinar
Webinar ID: 309-304-048
www.userzoom.com
First, the introductions…
Alfonso de la NuezPartner & Chief
Marketing Officerat UserZoom
Kim OslobResearch & Product
Strategy Directorat UserZoom
Bill AlbertDirector of the Design &
Usability Center, Bentley University
Alfonso
www.userzoom.com
Leading online user experience research software
company
Develops proprietary on-demand software app for online
research
Built for & by UX & marketing professionals
Offers a cost-effective, time-saving methodology
Has 9 years of experience in UX research & consulting
Has offices in Sunnyvale (CA), London (UK) & Barcelona
(Spain)
First, the introductions…
Alfonso
www.userzoom.com
What makes us proud?
Customers in 10 countries
Alfonso
www.userzoom.com
The Design and Usability Center at Bentley
University was founded in 1999
Provides UX consulting services to corporate clients
Supports the MS program in Human Factors in
Information Design
Focuses on user experience research and
evaluation
First, the introductions…
www.userzoom.com
Acknowledgements
6
Special thanks to Cynthia
Kamishlian and Dharmesh
Mistry from Bentley
University for carrying out
this research
Based on our recent book
“Beyond the Usability Lab”,
co-authored with Tom Tullis
and Donna Tedesco
www.userzoom.com
1. Introduction to both research methods (UZ, 10 mins)
2. The power of combining methods (DUC, 10 mins)
3. Case study: Parallel studies (DUC + UZ, 20 mins)
4. Conclusion: Looking at the ‘big picture’ (UZ, 10 mins)
5. Q&A (10 mins)
Agenda
Five things we’d like to talk to you about today:
Alfonso
www.userzoom.com
Brief introduction to both research methods
Usability testing in the Lab Moderation, think-aloud…
Bill
www.userzoom.com
Brief introduction to both research methods Usability testing in the Lab Observing, recording…
www.userzoom.com
Brief introduction to both research methods
Usability testing in the Lab Eye Tracking…
www.userzoom.com
Brief introduction to both research methods
Usability testing in the Lab Benefits/pros & limitations/cons
+ The face-to-face Catching the very details
+ Ability to ask questions/interact with the participant
live
+ The ability to invite others to attend the sessions
− Small sample size
− Lack of natural environment/context
− Cost +++
www.userzoom.com
Brief introduction to both research methods
Online Usability Testing
(a.k.a. Unmoderated Remote Usability Testing)
Kim
www.userzoom.com
Brief introduction to both research methods
Online usability testing At a glance…
• Hundreds of users can be tested
• Participation in the natural context…
• …from geographically spread locations
• No human moderation needed
• Our browser bar connects users with our secure servers
www.userzoom.com
Brief introduction to both research methods
Online usability testing Build the study first using a software solution
www.userzoom.com
Brief introduction to both research methods
Online usability testing Features! Power to the researcher
High-level of flexibility
Advanced task validation criteria (by question, by page interception, by time...)
Advanced questionnaires tool (all type of questions: Choice, scales, open...)
Insert multimedia (images, videos, flash, applets, sounds...)
Manage advanced logic/branching & conditions
Behavioral questionnaires (by interception)
Randomization: tasks, answers...
Piping capabilities
Mandatory options
Time-out options
Sample quality control
Multi-lingual studies support (including Asian languages).
…and more
www.userzoom.com
Partners of
Choose the user recruitment yourself
• Panel
• Intercept
• Private mailing list
Brief introduction to both research methods
www.userzoom.com
Brief introduction to both research methods
Online usability testing This is what it looks like…
www.userzoom.com
Brief introduction to both research methods
www.userzoom.com
Online usability testing Analyzing results
Brief introduction to both research methods
Effectiveness ratios
Efficiency ratios (time and clicks)
Visual clickstream (nav paths)
Click heatmaps
Users' suggestions & feedback
Satisfaction & brand perception indicators
Cluster analysis & dendograms
All on a per-task level, results gathered
automatically
www.userzoom.com
Online usability testing Analyzing results
Brief introduction to both research methods
www.userzoom.com
Brief introduction to both research methods
Online usability testing Benefits/pros and limitations/cons
+ Quantifying usability Objective, statistically significant
data
+ Automation of results, cost-effectiveness
+ Participation in the natural context
− Lack of face-to-face
− Can’t invite to observe sessions
− Must anticipate participant’s questions in advance
www.userzoom.com
1. Introduction to both research methods (UZ, 5 mins)
2. The power of combining methods (DUC, 10 mins)
3. Case study: Parallel studies (DUC + UZ, 20 mins)
4. Conclusion: Looking at the ‘big picture’ (UZ, 10 mins)
5. Q&A (10 mins)
Agenda
Five things we’d like to talk to you about today:
Alfonso
www.userzoom.com
Our UX Toolkit
23
www.userzoom.com
Power of Combining Methods
24
Every research method has unique strengths and
limitations
Gather new insights with each method
Greater confidence when observing similar findings
through multiple methods
www.userzoom.com
Seeing the Big Picture
25
www.userzoom.com
Which one goes first?
26
Lab first, then Online Online first, then Lab
Identify/fix “low hanging fruit”, then focus on remaining tasks with large sample size
Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues
Generate new concepts, ideas, questions through lab testing, then test/validate online
Collect video clips or more quotes of users to help bring metrics to life
Validate attitudes/preferences observed in lab testing
Gather all the metrics to validate design – if it tests well, then no need to bring users into the lab
www.userzoom.com
1. Introduction to both research methods (UZ, 5 mins)
2. The power of combining methods (DUC, 10 mins)
3. Case study: Parallel studies (DUC + UZ, 20 mins)
4. Conclusion: Looking at the ‘big picture’ (UZ, 10 mins)
5. Q&A (10 mins)
Agenda
Five things we’d like to talk to you about today:
Alfonso
www.userzoom.com
Case study: Parallel studies
We conducted 2 studies parallel. Here are the basic details:
• Website tested: Amazon.com’s Grocery and Gourmet Food
Store
• Lab study conducted by the DUC
• Online Usability Study conducted by UZ
• Date of the study: April 2010
www.userzoom.com
Case study: Parallel studies
Basic details of the studies (II)
• Objectives:
• What is the unique contribution of each method?
• What do we learn by combining methods?
• Participants: 10 for the lab study, 100 for the online study
• Study design:
• Initial questionnaire
• 3 tasks
• Follow up questions
• Final questionnaire
www.userzoom.com
Case study: Parallel studies
Participant Demographics from the Lab Usability Study:
• Average age 29
• Age range between 19 and 51
• 6 Males, 4 Females
• All had previously used Amazon.com website
• None had ordered groceries on-line
• Visited Amazon.com approximately every 2 weeks, on average
• Previously purchased books, electronics, toys, and more.
• Expected packaged, non-perishable grocery items to be sold on-line
• Most said they hadn’t bought groceries on-line because they, “like the experience
of picking up the produce and meat.”
• Expressed concern about products being fresh
• A few worried about exorbitant service or shipping charges making it not
worthwhile.
www.userzoom.com
Case study: Parallel studies
Task 1:
“You just ran out of dishwasher detergent. See if
Amazon has a pack of 6 Cascade Complete Dishwasher
Detergent Powder, in the 45 ounce size.”
www.userzoom.com
Case study: Parallel studies
Search vs. Menu Navigation• Approximately half of the participants wanted to immediately use SEARCH
Category Confusion• Most likely categories were “Home & Garden”, “Grocery, Health & Beauty”
• Chose “Grocery, Health & Beauty” through process of elimination
• Further confusion within “Grocery, Health & Beauty”
• “I don’t know if ‘Grocery and Gourmet food’ will have detergent”
Results of Locating Dishwashing Detergent
www.userzoom.com
Case study: Parallel studies
Confusion within “Grocery & Gourmet Food” menu• First menu is 4 screens long
• Participants were confused where to look for dishwasher detergent
Results of Locating Dishwashing Detergent
Dishwasher Detergent found under here
www.userzoom.com
Case study: Parallel studies
Nesting Confusion• All participants were confused by path to find dishwashing detergent:
• Amazon.com > Grocery & Gourmet Food > All Household (page is titled ‘Health & Personal Care ‘) > All Household Supplies > All Dishwashing > Scroll to locate Cascade Dishwashing Detergent
• Participants scrolled up and down navigation menus
• A few gave up mid-way & used SEARCH
“There are a lot of clicks to go through: In Household Cleaners, Kitchen Cleaners…I don’t see dishes…maybe by brand? I don’t see Cascade here. I would do SEARCH.”
“That took probably 5-6 minutes. For me, that’s really long.”
Results of Locating Dishwashing Detergent
www.userzoom.com
Case study: Parallel studies
Task 2:
“You want to buy Skippy Peanut Butter. Find out if there
are any discounts or coupons for it.”
www.userzoom.com
Case study: Parallel studies
Product Search vs. Coupon Search• A few looked on Amazon.com home page for Today’s Deals
• Most looked for the product in the “Grocery” section, and then looked for a coupon
• Some looked at “Special Offers” in the “Grocery” screen
• Special Offers led to a coupon and New and Used food, which disturbed participants.
Results of Locating Skippy Peanut Butter Coupon
www.userzoom.com 37
www.userzoom.com
Case study: Parallel studies
Participants that looked for the product complained about the length of the menu on the left side of the grocery page
• Peanut Butter, under “Sauces and Dips,” was four screens down plus 2 clicks. “It was tedious.”
Results of Locating Skippy Peanut Butter Coupon
www.userzoom.com
Case study: Parallel studies
Users had strong feelings about coupons for grocery items
“I would expect they’d show me the price they have, it doesn’t matter if it is cheaper.”
“If I am purchasing a big purchase, I would do it from Google. I would look for coupons [only] for big items.”
“I would have preferred to see a link from the home page– specials, coupons, etc…I don’t want to hunt down each thing.”
“It was not clear that a discount was available until you went to the [item] description page”
“I had to look through a lot of things."
Results of Locating Skippy Coupon
www.userzoom.com
Case study: Parallel studies
Task 3:
“Find out how you would schedule the same groceries to
be delivered every 3 months.”
www.userzoom.com
Case study: Parallel studies
All found the correct starting place: the “Grocery & Gourmet Food” page• Most noticed the “Shop Subscribe and Save” link
• Others tried to schedule each item from the item’s page
“I think you will have to do it for every item.”
“I would probably go to the product and check if I can schedule it.”
“[a product page] might say, “do you want this delivered?”
Scheduling Groceries for Repeated Delivery
www.userzoom.com
Case study: Parallel studies
Participants were confused on how to schedule grocery delivery• Most missed the delivery scheduling link at the top of the page
“The text was small, and I wasn’t sure I was in the right place.”
“The grayed-out part doesn’t jump out”
Scheduling Groceries for Repeated Delivery
DeliveryScheduling InformationLink
www.userzoom.com
Case study: Parallel studies
Confusing IA“Dishwasher detergent was under “Health [& Personal Care], which is not the first place I’d look.””
“There were too many categories, it confuses you…”
“It is not as well organized as other parts of Amazon.”
“I don’t know why ‘All Household Supplies’ is under ‘Health and Personal Care’”
Difficult navigation“Navigation wasn’t intuitive. The left-hand-side menu was so far down the page and not duplicated in the banner.”
“The browse list was too long. I don’t want to go through 10 chocolates before going to the pasta.”
“It didn’t feel intuitive.”
“I didn’t like how many levels I had to go through.”
“It was hard to find things. There were long lists to find food items.”
Conclusions from the Lab Study
www.userzoom.com
Case study: Parallel studies
Highlighted findings from the Online Usability Study:
Kim
www.userzoom.com
Study Background
45
Profile- 71% Male, 29% Female- Age range distributed between 18–55; 88% fell between 26-45-All have purchased from Amazon.com and visit Amazon once a week, month or every few months.- Eighty-eight percents of participants purchase groceries from in the store only, 6% online only and 6% both online and in a store.- Main reasons for considering purchasing groceries online is for convenience and time.
Details
- April 17th to April 26, 2010- Unmoderated remote method- 100 participants- Recruited through participant panel
www.userzoom.com
Tasks
1. Locate specific item: You just ran out of dishwasher detergent. Find Cascade Complete Dishwasher Detergent Powder, 45 oz (pack of 6). Validation by reaching the correct page with the detergent (URL validation).
2. Finding a discount code: You want to buy Skippy Peanut Butter with a discount you heard about on Amazon.com. Please write down the discount code. Validation by choosing the correct discount code (multiple choice).
3. Schedule grocery delivery: Find out how often you can schedule a delivery to your home. Validation by how often you can set up a delivery (multiple choice).
46
www.userzoom.com 47
Study Design
Initial Questionnaire:
How frequently do you visit Amazon.com?
Where do you usually buy your groceries?
Why have you considered purchasing groceries online?
Post Task:
Validation by url or question (Success, Error, Abandon ratios collected).
Ease of Use.
How intuitive links and menus were and how easy it was to start the search.
Issues and problems experienced while completing tasks.
Final Questionaire:
Overall satisfaction
Satisfaction with navigation, product search, information offered and look and feel of the site.
Likelihood to purchase groceries from Amazon.com.
Likelihood to recomment to a friend.
www.userzoom.com
T1: Locating Cascade Complete
48
SuccessNon-Success
Participants did feel particularly lost on the site (most likely due to experience using Amazon.com.
However they did not feel the menus and links were extremely intuitive and weren’t always sure were to start their search.
www.userzoom.com
T1: Locating Cascade Complete
49
Highlights dominant path
*3% insignificance removedOver half (51%) of the participants defaulted to using search indicating the menus and navigation of the site were not very intuitive.
Fifteen percent started in Health and Personal while 5% looked in promotional deals.
www.userzoom.com
T1: Locating Cascade Complete
50
What participants found difficult when attempting the task:
Exact Quotes:
“ Hard to navigate to find desired product. Too long and was getting frustrating.”
“ There were a lot of results so you have to type it in exactly or spend a couple minutes searching.”
“ I didn’t see a soap category and didn’t realize how the search function worked.”
“ I didn’t know where to look.”
www.userzoom.com
T2: Finding a discount code
51
SuccessNon-Success
Forty-seven percent of the participants thought it was very difficult to locate the discount code. They thought it was very unclear how to start searching, it was not where they expected it to be and were somewhat lost during the search process.
Extremely EasyExtremely Difficult
www.userzoom.com
T2: Finding a discount code
52
www.userzoom.com
T2: Finding a discount code
53
Error participants (55%) could not locate the code, thought it was too difficult to find the code or the peanut butter and stated:
“ I was expecting the discount in the results page, or the product page. I ended up looking for it under ’specials’, but I don’t know if that’s the best place for it.”
“ Too much searching and clicking.”
“ Looked under special offers having firstly used search, had to notice the offer code above product within the offer description.”
www.userzoom.com
T3: Schedule Grocery Delivery
54
SuccessNon-Success
Seventy-five percent of participants could not complete the task. They did not think it was clear how to start searching and felt lost during the search process. Forty percent of participants who were successful, felt it was very difficult to complete the task.
Extremely EasyExtremely Difficult
www.userzoom.com
T3: Schedule Grocery Delivery
55
Abandon participants (38%) could not find the information, thought the menus were difficult to understand and the task was taking too long.
“ It was not obvious to me that I needed to click on the link at the top right of the screen to get this information until I had spent quite a bit of time looking all over the rest of the screen.”
“ I wasn’t certain where to look for the information.”
“ Nothing about this process has been user friendly.”
www.userzoom.com
Final Questionnaire
56
Fifty-three percent of participants thought the navigation was very difficult, 39% thought it was difficult to search for a product, forty-seven percent rated the overall look and feel of the site as very poor.
www.userzoom.com
Final Questionnaire
57
Fifty-five percent of participants stated they would not purchase their groceries in the future from Amazon.com.
Net Promoter Score (NPS) calculation:0 to 6 = Detractors7 to 8 = Passive9 to 10 = PromotersNet Promoter Score (NPS) = % of Promoters - % of Detractors
www.userzoom.com
1. Introduction to both research methods (UZ, 15 mins)
2. The power of combining methods (DUC, 10 mins)
3. Practical example: Parallel studies (DUC + UZ, 20 mins)
4. Conclusion: Looking at the ‘big picture’ (UZ, 5
mins)
5. Q&A (10 mins)
Agenda
Five things we’d like to talk to you about today:
Alfonso
www.userzoom.com
Conclusions: Looking at the ‘big picture’
www.userzoom.com
Conclusions: Looking at the ‘big picture’
• Main issue, of course: Gourmet store built like the book
store
• IA + navigation + labeling problems encountered
• Both research methods prove this:
• By observing & listening at the Lab we learned a lot,
but then…
• …we also quantified how big the issues are through
metrics (UserZoom)
www.userzoom.com
1. Introduction to both research methods (UZ, 5 mins)
2. The power of combining methods (DUC, 10 mins)
3. Practical example: Parallel studies (DUC + UZ, 20 mins)
4. Conclusion: Looking at the ‘big picture’ (UZ, 5 mins)
5. Q&A Shoot! (10 mins)
Agenda
Five things we’d like to talk to you about today:
www.userzoom.com
º
62
USA440 N. Wolfe Rd.Sunnyvale, CA 94085 USA
Phone: +1 (408) 524 – 74 45Contact: Alfonso de la [email protected]
UKAylesbury House, 17 – 18Aylesbury Street, LondonECIR 0DB, UK
Phone: +44 (0) 20 7193 2171Cell: +44 (0) 7900 472 920Contact: Arthur [email protected]
SPAINAv. Diagonal 419 3º 2ª08008 Barcelona, Spain
Phone: +34 93 414 7554Cell: +34 93 368 4295Contact: Javier [email protected]
Thanks so much for your time!
Oh! Don’t forget: The slides and the video of this webinar will be available in a few days in our blog
www.userzoom.com