making the leap from web to mobile
DESCRIPTION
TRANSCRIPT
Making the Leap from Web to Mobile
Best Practices in Mobile User Experience Research
Amy Buckner, AnswerLab Kris Mihalic, Yahoo!
UPA 2009, June 12, Portland
3
CONFIDENTIAL
Agenda
Topic Minutes
1. Introduction 5 minutes
2. State of Mobile 10 minutes
3. Group Exercise: Mobile Device Task & Discussion 5 minutes
4. Mobile User Research Methods 20 minutes
5. Team Exercise: Develop Research Solution 15 minutes
6. Case Study 15 minutes
7. Mobile Usability Research Challenges & Tips 15 minutes
8. Q&A 5 minutes
4
CONFIDENTIAL
Introductions
Amy Buckner Managing Partner & Co-Founder of AnswerLab Over 10 years experience leading user research programs for Fortune 500 companies
Expertise in quantitative and qualitative methods, across web and mobile devices
Previous West Coast Director of Professional Services at Vividence (now Keynote) and Marketing Manager at SmartPlanet
Believes a glass of wine and a beef jerky stick count as dinner
Kris Mihalic, PhD Head of Mobile User Experience and Design Research, Yahoo! Seasoned user experience and design researcher for mobile and cross-channel products
and services
Expertise in defining and executing UX and design research strategy in cross-functional teams
Experience teaching interface design methods, usability engineering, and telecommunication technologies
Lives on coffee
5
CONFIDENTIAL
State of Mobile
6
CONFIDENTIAL
There are mobile phone subscribers worldwide
Only of US mobile phones are iPhones (~5 Million)
Social networking on mobile phones is growing Y/Y
There are more mobile IM users than mobile business email users in the US
4.1 Billion
2%
196%
1.4x
7
CONFIDENTIAL
Mobile Market Summary
Source: Experian 2008, eMarketer 2008
Globally, there are 800 million cars, 850 million PCs, 1.3 billion fixed landline phones, 1.4 billion credit cards, 1.5 billion TV sets – and 2.7 billion mobile phones – in use. (Experian 2008)
eMarketer projects that worldwide spending on mobile advertising will reach a total of $19.1 billion in 2012 – up from 4.6 billion in 2008.
8
CONFIDENTIAL
U.S. Mobile Web Users Growing Rapidly
Source: m:metrics/comScore, 2009
9
CONFIDENTIAL
Mobile Audience Size (# of users)
52%
31% 30%
10%
SMS Users Mobile Media Users 3G Smartphone 0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
In the U.S., the size of the Mobile Audience is 250 Million
Source: m:metrics/comScore, 2009
10
CONFIDENTIAL
U.S. Mobile Internet Usage
Mobile Internet users are heavy users
60% access the Internet from their phone at least once/day
48% access the Internet more than once per day
Source: TSM|TargetProfile 2007
11
CONFIDENTIAL
Device Traffic
Mobile phones dominate mobile Web traffic with 66% market share (US).
BlackBerry and Smartphones (Windows Mobile, Palm, etc.) are heavy users of mobile Internet – combined 26% of traffic
iPhone/iPod Touch has shown tremendous growth, capturing 8% of the mobile Internet market in the US
Source: m:metrics/comScore, 2009
12
CONFIDENTIAL
Widgets Improve Mobile Internet Experience
Source: ‘Widgets Improve the Convenience of the Mobile Internet,’ Forrester, 2008
13
CONFIDENTIAL
Group Exercise
14
CONFIDENTIAL
Take out your cell phone and search for times that ‘Star Trek’ is
playing in your neighborhood.
Raise your hand when you have finished the task.
15
CONFIDENTIAL
Discussion Points
How long did it take?
How did you do it? (web, app, SMS?)
What problems, if any, did you have?
16
CONFIDENTIAL
Mobile User Research Methods
17
CONFIDENTIAL
Overview of Research Methods
Quick and Dirty feedback
Lab Studies
Field Studies
Online Surveys Basic Survey Survey with Behavioral Tracking (Keynote on iPhone only)
Quantitative Behavioral Analysis
18
CONFIDENTIAL
Quick and Dirty – What are obvious problems?
Advantages
Informal setting with Lo-Fi prototypes
Fast iteration cycles
Brain-damage check
Finds critical usability issues in shortest time
Challenges
Insider/stakeholder view
Anecdotal, unstructured results
Small sample size
Difficult to convey real usage scenarios
Examples
Cross-functional team feedback
Internal alpha/beta
19
CONFIDENTIAL
Lab Studies – Can they use it?
Advantages
Allows for probing by moderator
Can record interactions
Best for prototype testing
Allows for real-time viewing of interaction – team engagement
Challenges
Artificial environment
Moderator bias; participant ‘pleasing’
Small sample size
Difficult to test all hardware / network scenarios
Examples
Usability study
Participatory design sessions
20
CONFIDENTIAL
Field Studies – What is the context of usage?
Advantages
Assess usage under real conditions
Candid user feedback
Delivers unknown use-cases
Challenges
Small sample size
Difficult to observe and probe
Participant engagement difficult
Requires robust product
Examples
Diary study
Community study
21
CONFIDENTIAL
Online Surveys – What do they think of it?
Advantages
Can deliver statistically valid results
Provides qualitative and/or quantitative data
Geographical diversity (remote)
Broad representation of devices
Challenges
Difficult to observe and probe
Out-of-context interaction, esp. with desktop survey
Technical constraints
Examples
Desktop surveys
On-device survey
Survey with behavioral tracking
22
CONFIDENTIAL
Behavioral Analysis – How do they use it?
Advantages
Reflects what people do, not say
Statistically valid results
Shows all behaviors, rather than those confined to a single task
Challenges
Probing / deep-dive difficult
Context and intent unknown
Attitudes and perceptions unknown
Examples
Log data mining
23
CONFIDENTIAL
Types of Prototypes
Paper prototypes Mocks
Paper and pencil
Interactive prototypes Apps
Flash
Native, e.g. iPhone
Browser-based Simple HTML prototype
High-fidelity prototype (with session variables)
24
CONFIDENTIAL
Team Exercise & Case Study
25
CONFIDENTIAL
Yahoo! Go
Mobile Application Search News Sports Weather Other Topics
26
CONFIDENTIAL
Y! oneSearch integrated into Y! Go
27
CONFIDENTIAL
Business Issue
Drive mobile search uptake – Improved user experience can help accelerate growth of the service
Improve relevancy – Understanding search intent can contribute to delivering more relevant results
Product differentiation – New approach to search results page can deliver higher value to customers
28
CONFIDENTIAL
Research Objectives
Understand mobile search behavior in users’ daily lives
Identify content users seek when conducting mobile searches
Assess the context surrounding mobile searches
Evaluate effectiveness of new product
Identify opportunities to improve the user experience
29
CONFIDENTIAL
1. Form into teams of 4.
2. Develop a research plan for Yahoo! Go.
3. You have 10 minutes.
30
CONFIDENTIAL
Understand mobile search behavior in users’ daily lives
Identify content users seek when conducting mobile searches
Assess the context surrounding mobile searches
Evaluate effectiveness of new product
Identify opportunities to improve the user experience
Research Objectives
Develop a research plan for Yahoo! Go
Include: • Recommended method(s) • Number of participants • Profile of participants
Team Activity
31
CONFIDENTIAL
Discussion Points
What methods do you recommend & why?
What are your anticipated concerns?
Any potential limitations?
32
CONFIDENTIAL
Our Solution: One-Month Field Study (1)
Voicemails & Pocket Card
9 users
Daily voicemails
Laminated pocket card with key questions
Daily Mobile Survey
Daily SMS mini-survey
Linked to phone number for voicemail
Digital Photos
Users sent photos of themselves or surroundings in the context of using oneSearch
33
CONFIDENTIAL
‘Pocket Card’ Questionnaire
34
CONFIDENTIAL
Sample Voicemail Diary Messages -- Winchelle
Friday 7:39pm I used oneSearch today, was looking for a hospital – directions and address to it. I was getting off work and sitting in my car, trying to find directions. It was about 4:30. I liked that it came up with the website and with information about that. And the directions on how to go. I didn’t like that it could not find my work address – which is saved on my Yahoo! directions (recently gone) saved places. I think using the actual internet through the PC will help me better because it knows where the address is. I’m not sure why the address is not coming up in my oneSearch and that sucked.
Saturday 8:06pm I didn’t use oneSearch today. I looked for info, looked for a store and I was at home. I wanted to know if the store was open STORE HOURS (9:45am). I used the internet with my phone and used Google to search. It was not very helpful because I wanted to see what time they opened and I couldn’t find it. I’ve used oneSearch in the past and I guess the results were the same but I guess it’d be nice to go to the site for that particular store.
Monday 10:48pm I used oneSearch to find Expedia, was at work on my lunch break. It was about 12:20. I liked that I can find it quickly. I guess the website didn’t support my mobile phone though, so I didn’t like that. Would rather book a flight/hotel at home because its supported.
35
CONFIDENTIAL
Our Solution: One-Month Field Study (2)
Pre-Interview
An initial 45-minute interview
Verified technical capabilities of phone
Explained research program
Mid-Check Interview
20-minute check-in phone interview
Initial feedback
Q&A
Wrap-up Interview
Final in-person interview of 75 minutes
Clarified voicemail reports
Overall impressions of the product from the month-long usage
36
CONFIDENTIAL
Screening Criteria: 9 Experienced Mobile Users
Have an unlimited data plan for WAP services through their mobile carrier
Use their mobile for text messages (SMS) daily and use mobile more than three times per week for activities that require the transmission of online data
Have Cingular, Sprint, T-Mobile, or Verizon as their mobile carrier
Have a primary mobile device capable of downloading and using Y! Go
Live or work within San Francisco
Mix of gender, age (23-46+), and typical method of commuting to work (4 users primarily drive, 5 primarily take mass transit)
37
CONFIDENTIAL
Sliding Scale for Participant Incentives
$100 for the initial in-person interview
$50 for the mid-way phone interview
$150 for the final in-person interview
$3 each day a report is given
38
CONFIDENTIAL
Natural environment key to context
All research activities on the mobile device
Mix of methods Engage users in program Breadth of insights – context, usability, impressions
Why did we choose this method?
39
CONFIDENTIAL
What did we learn? (1)
Perceived value increased with usage
Usage typically driven by lack of computer availability
However, certain scenarios drove phone choice over computer:
Social setting Privacy Convenience
40
CONFIDENTIAL
What did we learn? (2)
Mental model based on computer usage
Speed and relevance of search results highest area of frustration
Lack of understanding that results were customized by widget
41
CONFIDENTIAL
Daily SMS Survey Results
*Website not adapted for mobile device
42
CONFIDENTIAL
What did we learn? (3)
Use Case Primary Content Wanted
Shopping Business’ number, location, directions, and hours of operation
Entertaining Schedules/Venues of concerts, sport games, movies, and TV shows
Researching Content from news/articles
Dining Restaurants’ number, location, directions, and hours of operation
Traveling Schedules and numbers of transportation (public transit, taxis, etc)
Media Seeking Photos/Ring tones
43
CONFIDENTIAL
Summary of Use Case Frequency
Note: The 166 searches were conducted by 9 users
44
CONFIDENTIAL
Business’ number, location, directions, and hours of operation
“While driving home from work, I did a search for Fry’s electronics on the local search and it came up with all the different Fry’s in the area, their numbers, as well as directions and all that. The results were all helpful and fast in helping me find the one closest.”
“Since we were already out shopping, I wanted to find where the closest Citibank was to where we were.”
Pricing/Product comparisons “I was at Costco and comparing TV prices. I
wanted to know how much these same TV’s would be at the Circuit City down the street.”
Shopping
45
CONFIDENTIAL
A restaurant’s number, location, directions, and hours of operation “I just woke up and didn’t want to boot my
computer up but I wanted to find out what time this breakfast place opened so I could know how soon I could get there.”
“I was in my car and wanted their number so I could call and place an order to pick it up on my way home from work.”
Ideas of places to eat “I was with my wife in the Haight and we
wanted to find nearby restaurants so we could decide what we wanted to eat that was within walking distance.”
“I was in the South Bay with some friends and we wanted to find a good sushi place nearby.”
Dining
Reservations & Reviews “I was hoping to make reservations straight
from the phone, or at least call them to place them.”
“I was out with some friends and curious about whether the place nearby was any good.”
46
CONFIDENTIAL
Transit schedules and numbers “I wanted to find out when the next MUNI but
the MUNI’s site wouldn’t work on my phone.” “I wanted the number to call a taxi. I was
outside and tired of waiting for the bus.”
Tourist activities “I wanted to see what kind of ferry tours we
could do before looking for another museum that would be fun for my daughter.”
“I was looking for fun things to do while at Hermosa beach.”
Maps and directions “I was trying to get a map of the Mt. Shasta area
and I wanted to plan which route to take. It would be nice if the driving directions had live updates on traffic and road conditions.”
Flight status / check-in “I was in the taxi to the airport and wanted to
check my flight’s status.”
Traveling
48
CONFIDENTIAL
Business Implications
Drive mobile search uptake – Distribution through partnerships, marketing campaigns, opening to third party services and developers
Improve relevancy – Focused on improving results for specific use cases (e.g. local businesses, flights, etc.)
Product differentiation – Federated search results, integrated user experience across multiple services (e.g. search results + maps)
49
CONFIDENTIAL
Mobile Usability Research Challenges & Tips
50
CONFIDENTIAL
Variety of Mobile Devices
Difficult to know and understand user experience across all devices
Devices vary by model, browser type, carrier, and input type
Moderator may be unable to help users through usability tasks – resetting prototypes, navigating – if interface is not well-known
Design team should have a variety of devices simulators / devices on hand for testing
Prototype designs should be tested across as many devices as possible
Limit recruiting to only devices that have been tested
Have a back-up plan with either a basic device or paper prototypes
Challenges Tips
51
CONFIDENTIAL
Challenges Tips
Mobile Devices are Small and . . . Well, Very Mobile
Users move the device around while interacting and explaining, often moving video display out of focus
Reflectors on screens and smudges can make video all reflection
Small device screens make it difficult to see what users are doing
Large fingers and long fingernails can cause unintended device responses
Create a hot zone on the table, encouraging users to keep the device within a narrow frame
Consider light source and potential reflection on the phone; continually adjust mid testing; turn-off lights
Use remote-controlled video camera (with technician in back room) or have a second technician available in interview room
Project the mobile screen on a larger monitor in interview room and back room
Creatively screen out participants whose finger size may distract from findings (if non-touch screen)
52
CONFIDENTIAL
Challenges Tips
Varied Locations and Use Cases
Mobile device usage occurs in a variety of places 59% of Americans check email while in
the bathroom*
Difficult to recreate the true experience in a lab setting
Create opportunities for feedback in context of usage
Conduct field studies on beta products
MOBILE DEVICE ACCESS
Source: Nielsen Mobile 2008
55.2%
35.9% 15.8%
67.4%
6.4%
53
CONFIDENTIAL
Challenges Tips
Varied Carrier Network Coverage
Network coverage is inconsistent; difficult to predict accessibility during lab testing
Difficult to validate that out-of-town lab truly has adequate coverage
Visit facility with colleagues who have various carriers to confirm network availability in lab
From out-of-town labs, request: Reference clients from last mobile
studies List of carriers confirmed to have
consistent coverage At a minimum, check carrier
coverage in various cities online
54
CONFIDENTIAL
Challenges Tips
Calling / Data Plans Vary Widely
Participants may be charged for mobile web access or sample texts sent during testing
Participants may change data plan between recruiting interview and date of study
Screen during recruiting for: Unlimited data plans Willingness to send / receive text
messages during study
Rescreen again upon arrival for study
55
CONFIDENTIAL
Challenges Tips
Varied User Experience
Most mobile device owners only utilize a small percentage of total device capabilities 43% of mobile subscribers do not use
text messaging on a regular basis*
Users may have no experience typing in a web address to access a site or with sending SMS messages (which may be critical to your study)
If using a prototype phone for downloadable app, users may not be comfortable with it
Pre-test participants to ensure appropriate device experience Mobile address typing Send SMS to recruiter
Spend 2-5 minutes explaining how prototype phone works
* Source: Nielsen 2008 ‘The Short Code Marketing Opportunity’
56
CONFIDENTIAL
Other Helpful Tips
Tell participants to bring recharger
Simplify language (e.g., SMS vs. text message)
Have a prototype strategy Back-end SMS text simulation Easy URL for access Back-up paper prototypes Index page for easy access to alternative flows Have plan for interruption from phone call or text (resetting session variables)
Plan for getting screenshots for your report iPhone: Press hold button and home button at same time (saves into photo gallery) Use documented comps
57
CONFIDENTIAL
Q&A
58
CONFIDENTIAL
Thank You.
Amy Buckner Managing Partner & Co-Founder, AnswerLab [email protected]
Kris Mihalic Head of Mobile User Experience and Design Research, Yahoo! [email protected] Twitter: suikris