beyond usability testing: getting a holistic view of the user experience
DESCRIPTION
Talk given to Atlanta Web Design Group, August 2013TRANSCRIPT
![Page 1: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/1.jpg)
+Beyond usability testingGetting a holistic view of the user experience
Amanda Nance, User experience researcher
Atlanta Web Design GroupAug. 29, 2013
![Page 2: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/2.jpg)
+About me
In UX field since 2002
Studied human-centered communication & psychology, Mercer University
M.S. in Human-computer interaction, Georgia Tech
Women’s barbershop quartet
@amandaux #beyondutest
![Page 3: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/3.jpg)
+Context
Large, international corporation (14,000 employees)
Accounting products for small businesses Desktop and web-based No analytics
My product Product Management team - 5 Developers - 16 Dev Operations - 5 QA - 6 Interaction & visual designers - 3 Writers - 2 part-time Researcher - 1
![Page 4: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/4.jpg)
+Context
31 studies since Jan 2010
120+ users
![Page 5: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/5.jpg)
+About you
![Page 6: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/6.jpg)
+My goal
Broaden your toolkit for including users
Convince you to start right away
![Page 7: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/7.jpg)
+Usability testing is great. Try it!
See “Rocket Surgery Made Easy” by Steve Krug
![Page 8: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/8.jpg)
+An ideal usability test would tell you everything, but…
Ideal In practice
Real users yes!
Identify unknown needs
Natural environment
User's own data
Real motivation debatable
Non-core tasks
Major usability problems apparent yes!
Minor usability problems apparent yes!
Quantitative information sometimes
![Page 9: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/9.jpg)
+So we have these other methods
![Page 10: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/10.jpg)
+
Field visits
![Page 11: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/11.jpg)
+Watch and interview users in their natural habit
![Page 12: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/12.jpg)
+Use field visits before designing to learn what people need
Before designing can mean: Before anything Before finalizing the design After launch to improve product
![Page 13: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/13.jpg)
+
Used under CC licenseLeah Gregg
Asking users about problems limits the problems you’ll uncover
![Page 14: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/14.jpg)
+The problem we solved was one users hadn’t thought about
“I hadn’t even thought that far out of the box.”-User reaction to our product concept
![Page 15: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/15.jpg)
+Observation gives you details about what people need
![Page 16: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/16.jpg)
+Use field visits for learning about sensitive topics
![Page 17: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/17.jpg)
+Tips: Before the visit
Ask about pets
Talk to the participant yourself before the visit Even if using an external recruiter Ask to save up some work Tell them not to clean up!
Ask for driving directions / verify Google Maps accuracy
![Page 18: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/18.jpg)
+Tips: During the visit
Say “Show me” and ask for specific examples
![Page 19: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/19.jpg)
+Tips: During the visit
Verify your understanding in the moment
Always respond professionally
![Page 20: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/20.jpg)
+Field visits: general guidelines
We pay $300 for 3 hours
Limit is 3 hours
Can divide into one-hour sessions amongst 3 users
![Page 21: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/21.jpg)
+
Informal interviewsCalls & emails, screen sharing
![Page 22: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/22.jpg)
+Use informal interviews for (perceived) requirements
Email and phone calls: easy and quick!
![Page 23: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/23.jpg)
+Use screen sharing interviews to learn how the product is (really) used
We didn’t account for users who repeatedly subcontract for the same company.
![Page 24: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/24.jpg)
+Screen sharing tips: recruiting
Find customers who are interested in the topic Analytics Survey responses Email
Explain logistics: ear buds or speakerphone required
Get participant’s phone number
![Page 25: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/25.jpg)
+Screen sharing tips: tools
Join.Me - recommended by colleague
![Page 26: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/26.jpg)
+Screen sharing tips: tools
Turn off the beeps when people join
Turn off chat for observers
Turn off list of meeting attendees
![Page 27: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/27.jpg)
+Screen sharing tips: during the session
“Can you show me…” the issues you’re having how you _______
Help with issues at end
In case of emergency: let participant control your screen
![Page 28: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/28.jpg)
+Screen sharing: general guidelines
We pay $75/hour
$25 for 20-30 min
Limit to 1 hour
![Page 29: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/29.jpg)
+
Live recruiting testAKA intercept test
![Page 30: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/30.jpg)
+Live recruit test = watching more natural behavior
CC licensehttp://www.flickr.com/photos/evilpeacock/6423510787/
![Page 31: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/31.jpg)
+Use a live recruit test when motivation is key
Pretend that you are looking for bookkeeping software for your company…
![Page 32: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/32.jpg)
+Live recruiting tips: Ethnio
Use Ethnio to recruit
Read all demographics (unqualified people are listed)
Answers may reveal unclear screener questions
Pause the recruiting if you leave
![Page 33: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/33.jpg)
+Live recruiting tips
Decide how to handle people who have already started using the site
Have sales or relevant phone numbers handy
See www.remoteresear.ch for templates
![Page 34: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/34.jpg)
+
Surveys
![Page 35: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/35.jpg)
+Surveys
Questionnaire completed by users to learn about demographics, opinions, etc.
Use when you need quantitative data
![Page 36: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/36.jpg)
+TipsFOCUS Keep it short
CC licensehttp://www.flickr.com/photos/44456430@N04/6825380102/
![Page 37: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/37.jpg)
+A goal-less survey
1. How did you first become aware of Awesome Accounting?
2. To the best of your knowledge, what does Awesome Accounting do?
3. To the best of your knowledge, what is the price of Awesome Accounting?
4. What software were you using in the running of your business when you first considered Awesome Accounting?
5. What software were you looking to replace when you first considered Awesome Accounting, if any?
6. Why did you first consider Awesome Accounting?
7. Before subscribing to Awesome Accounting, did you do a free trial?
8. Are you currently using Awesome Accounting in running your business?
9. Which of the following parts of Awesome Accounting have you used, or do you intend to use, in running your business?
10. How satisfied are you with the following aspects of Awesome Accounting?
11. Please use the spaces below to comment on each of the aspects of Awesome Accounting.
12. How likely are you to continue to be a Awesome Accounting subscriber 12 months from now?
![Page 38: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/38.jpg)
Used with CC permissionhttp://www.flickr.com/photos/stovak/
![Page 39: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/39.jpg)
+Tip: Survey process
Ask questions that people can accurately answer What did you eat for breakfast 4 days ago? vs. Do you typically organize your emails into folders?
Include an “N/A” option
Avoid mandatory questions
Ask permission to contact for follow-up questions
Pilot test your survey
![Page 40: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/40.jpg)
+
Diary studies
![Page 41: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/41.jpg)
+Users record activities, problems, or feelings over time
CC license http://www.flickr.com/photos/bitchcakes/5457865373/sizes/o/in/photostream/
![Page 42: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/42.jpg)
+Diary studies: when to use
Seeing experience over time Researching and buying a car
Infrequent tasks / non-core tasks
To identify places to dig deeper with other studies
![Page 43: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/43.jpg)
+Diary studies will show you the bigger issues
![Page 44: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/44.jpg)
+Diary studies: Tips
Reserve time to follow up on unclear entries
Interact with participants to maintain engagement
Continue participants who are useful
Ask for screenshots
![Page 45: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/45.jpg)
+Diary studies: general guidelines
Compensation At least $25/week Can increase over time
Length: Can keep people engaged for several weeks if not months
Recruit: At least 10 More depending on your goals
![Page 46: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/46.jpg)
+
General research tips
![Page 47: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/47.jpg)
+General research tips: planning and logistics
Start with goals
Pilot
Consider recording with permission
Have someone else take notes if possible
![Page 48: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/48.jpg)
+General research tips: facilitating
Ask open-ended questions
Ask neutral questions
Probe for more information
Ask for an example
Be genuine
Show that you are on the user’s side
![Page 49: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/49.jpg)
+
How to get started
![Page 50: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/50.jpg)
+What is keeping you from getting started?
![Page 51: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/51.jpg)
+Getting started
Start lean and build up
Ask forgiveness, not permission from stakeholders
Find users who will help for free
Get your stakeholders involved
Don’t worry about being a research expert
![Page 52: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/52.jpg)
+
More infoBooks, links, and stuff
![Page 53: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/53.jpg)
+Quick read to get started
UX for Lean Startups: Faster, Smarter User Experience Research and Designby Laura Klein
Short. No UX background required.
![Page 54: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/54.jpg)
+General references on qualitative research
Observing the User Experience, Second EditionElizabeth Goodman, Mike Kuniavsky, Andrea Moed
500+ pages. Each method = 1 chapter
Understanding Your Users: A practical guide to user requirementsby Catherine Courage & Kathy Baxter
700+ pages. Each method = 1 chapter.
![Page 55: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/55.jpg)
+On field visits
User and task analysis by Hackos and Redish
Contextual Design by Beyer and Holtzblatt
![Page 56: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/56.jpg)
+Live recruit testing and remote research
Remote Research: Real Users, Real Time, Real Researchby Nate Bolt, Tony Tulathimutte
The general reference books I mentioned don’t seem to cover this topic.
Look at www.remoteresear.ch for templates
![Page 57: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/57.jpg)
+On interviewing
Steve Portigal’s “Interviewing users”
![Page 58: Beyond usability testing: Getting a holistic view of the user experience](https://reader034.vdocument.in/reader034/viewer/2022051613/54c7321b4a7959e7698b464d/html5/thumbnails/58.jpg)
+On surveys
Coming in 2014: “Surveys that work” by Caroline Jarrett
Caroline’s podcast with UIE on surveys