overseas usability testing
DESCRIPTION
NOTE: SOME OF THE SLIDES IN THIS ONLINE SLIDESHOW ARE DEFECTIVE. THE DOWNLOAD VERSION IS FINE. Are you planning to conduct usability tests in other countries? It is more challenging (and rewarding) than you might think. Here is a case study of usability tests conducted in three European markets that highlights what worked — and what did not. This unique case study explores what happens when you drive a test with the following statistics: 12 days 10,723 miles in travel 3 testing locations 3 third-party agencies 1 prototype 3 languages 24 test participants Sound like fun? Read on.TRANSCRIPT
A Case study
BY JOE DYER
Overseas usability testing
@joeplanet
I had the remarkable experience of managing the usability tes;ng of a prototype in Europe across three markets.
A long time ago in a country far, far away...
★ 12 days
★ 10,723 miles
★ 3 tes;ng loca;ons
★ 1 prototype
★ 3 languages
★ 24 test par;cipants
by the numbers
★ 12 days
★ 10,723 miles
★ 3 tes;ng loca;ons
★ 1 prototype
★ 3 languages
★ 24 test par;cipants
by the numbers
LONDON
PARIS
MILAN
Who is this for?
usability experts
Business orproduct managers
Who is this for?
the morbidly curious
Who is this for?
wait, what are we talking about?
the morbidly curious
Who is this for?
is testing Different overseas?
How testing is different overseas
INTERACTIVE AGENCY
Tes;ng abroad almost always involves working with a tes;ng agency. Think expensive contracts.
How testing is different overseas
Less than favorable exchange rates in the last several years
How testing is different overseas
London
milan
Have you seen the cost of interna;onal travel these days?
How testing is different overseas
votre haleine sent l'ail!
He says, your breath smells like garlic!
Transla;on services are likely to be an addi;onal cost.
He says, your breath smells like garlic!
4
what worked well and what did not?
preparation facilitation execution
Test objec;ves and scenarios were wriRen internally before shared with tes;ng consultants.
Plan early & Share early
Number of test objec;ves in the usability test plan were too numerous for the test as it was executed.
appropriate # of tasks
Facilitators wasted too much ;me on unrequested benchmark tes;ng ques;ons.
appropriate # of tasks
agency client
Maybe two conference calls and a dozen or so emails were not quite enough.
Engage early & often
ZZZZ
agency client
Engage early & often
Maybe two conference calls and a dozen or so emails were not quite enough.
Test preparation
Do all that you can to assess the skills and abili;es of any agency or contractor you intend to hire.
Get mul;ple bids if you can.
Over communicate all tes;ng objec;ves un;l they are capable of being your proxy.
We created a single prototype, in three languages.
the prototype
Limited sta;c HTML model made it nearly impossible for test par;cipants to explore the possibili;es of the prototype.
go high-fidelity
Visitors are expected to
complain at the office between
the hours of 9 and 11 am daily
Some translated words were too literal and were not reviewed for context within the prototype.
lost in translation
Please leave your values at the front desk
The genuine antics in your room come from our family castle. Long life to it.
special today: no ice cream
our wines leave
you nothing to
hope for
special today: no ice cream
We take your bags and send them in all directions
The manager has
personally
passed all the
water served here
We take your bags and send them in all directions
Come alive with the Pepsi GenerationCome alive with the Pepsi Generation
Pepsi will bring your ancestors back from the dead
Pepsi will bring your ancestors back from the dead
Salem - Feeling FreeSalem - Feeling Free
When smoking Salem, you feel so refreshed that your mind seems to be free and empty
When smoking Salem, you feel so refreshed that your mind seems to be free and empty
selecting test participants
The French screener?
The Italian screener?
The English call screener was easy to review and approve.
Scenarios were given to par;cipants wriRen on a card for consistency.
printed test scenarios
Facilitators were not familiar with the prototype or test objec;ves prior to tes;ng.
and even more issues ...
keeping it on the fairway
How am i doing?
keeping it on the fairway
How am i doing?
do you really need me to tell
you again?
keeping it on the fairway
Instant Messenger can be your friend.
test facilitation
the domino effect
Long process of contrac;ng, legal
review, etc
the domino effect
Test plan covered in two conference calls
the domino effect
Long process of contrac;ng, legal
review, etc
Test plan covered in two conference calls
Benchmark tes;ng
encroached on scenarios
the domino effect
Long process of contrac;ng, legal
review, etc
Test plan covered in two conference calls
Benchmark tes;ng
encroached on scenarios
Facilitator’s lack of familiarity with prototype
the domino effect
Long process of contrac;ng, legal
review, etc
draft report
final report
beware expensive reports
so, why three cities at nearly the same time?
MARKET COMPARISON
LARGE medium SMALL
underscore user differences
MARKET COMPARISON
LARGE medium SMALL
i 1. Very internet savvy2. High rate of adoption3. Not likely to need offline help4. market is ready for the solution
i
underscore user differences
MARKET COMPARISON
LARGE medium SMALL
i 1. more market potential2. not likely to need offline help3. solution has a chance to succeed with some modifications
i
underscore user differences
MARKET COMPARISON
LARGE medium SMALL
i 1. catastrophic usability findings2. Lower rate of adoption3. most likely to need offline help4. solution doesn't work in this locale
i
underscore user differences
reduced travel costs
reduced travel costs
Key business managers were present during the tests, directly observing poten;al customers interac;ng with the new e-‐commerce solu;on.
so biz managers could see market reactions
the time machine
just getting to work
the time machine
last test of the day
the time machine
just getting to work
where to eat dinner?
the time machine
last test of the day
just getting to work
All resources working on the prototypes should not be on vacation or tied up with other commitments during the time of testing.
Consider moving to an alternate schedule.
this is no time for a vacation
0:00 / 4:59
Sorry. The streaming video has timed out. See your System Administrator for help.!
technical difficulties
Daily conference calls were setup to discuss test findings and build consensus on the severity of each finding.
daily conference calls
allow for travel time
it's just a browser, right?
4 ways to test overseas
4 ways to test overseas
plan it all and hire help
keynote (formerly vividence) study
4 ways to test overseas
4 ways to test abroad
outsource it completely
4 ways to test overseas
4 ways to test abroad
rogue or ad hoc testing
4 ways to test overseas
and so, in conclusion ...
OVERSEAS TESTING PRE-‐FLIGHT CHECKLIST
questions?questions?
iStockphoto
Flickr
UCD Stencil by Zazelenchuk & Boling
Glyphish Icons
My Wacom tablet and two hands
resources
Created while listening to Master of Puppetsby Metallica and Are You Experienced by Jimi Hendrix