international business user research: methods and tools
DESCRIPTION
Qualitative Research on International Business Users How to interview (business) users Cross-cultural user research Online (remote) User Research Mouse tracking heatmap Remote user testing User feedback systemTRANSCRIPT
International Business User Research:Methods and Tools
Zhang, Yong
About me
张勇 （ Zhang ， Yong ） Vanderbilt University
M.S. Management of Technology
Alibaba.com User Research / Data
Analysis 5 yrs of business user research
covering both International Buyers and China Suppliers
Explored and implemented various user research methods and tools: Online Survey, Remote User Testing, Focus Group, Interview, AB-Testing, Persona and other Data mining projects
Contents
Qualitative Research on International Business Users • How to interview (business) users• Cross-cultural user research
Online (remote) User Research• Mouse tracking heatmap• Remote user testing• User feedback system
Planning Requirements Design Implementation Test & Measure Post Release
Lead user study User surveys Design guidelines Style guides Performance testing Data analysis
Stakeholder meeting(Brainstorming)
Interviews Paper prototyping Rapid prototyping Diagnostic evaluation Subjective assessment
Competitor analysis Contextual inquiry Heuristic evaluation Cognitive walkthrough User surveys
Data analysis Field study User observation
Parallel design A/B testing Remote evaluation
Secondary research Focus group Storyboarding
Persona Wizard of Oz
Evaluating existing system
Interface design patterns
Card sorting
Affinity diagramming
Scenarios of use(User diary)
Task analysis
User research methods in product development lifecycle: Quantitative – Qualitative – Quantitative
Concept FBRD Specs Checklist Review
（ from The Persona Lifecycle: Keeping People in Mind Throughout Product Design ）
More (advanced) methods and tools
All pictures from Google image search
But
If most of your users are international
and your only have 2 weeks, at mostcannot find users to come to your lab
……
As of December 31, 2012, the platform had around 36.7 million registered users from more than 240 countries and regions and showcased more than 2.8 million supplier storefronts.
http://news.alibaba.com/specials/aboutalibaba/aligroup/index.html
You may want to consider: Online (Remote) User Research
Instant Message/Phone Interview
Instant Message/Online Chat Room/Conference Call Focus Group
http://www.slideshare.net/boltpeters/remote-research-at-ixd10
How to choose from the Remote User Research Toolbox
AutomatedSimple Tasks
A/B Tests
Various FormsOf Online Surveys
Mouse TrackingHeatmap
Online Focus Group or Phone/IM interviews
RemoteUser Testing
Project Progress
Most popular user research methods
Methods Solve N Min. Time Pros Cons
Online Interview / Focus Group
Explore concept & solutions
6-12 6 ~10 daysRich insight, in
depth
Appointment for Int’l Focus
Group
Remote user testing
Specific interaction solutions
6-12 6 ~10 days
Effective in finding
usability problems
Appointment for Int’l Focus
Group
Online tests: (card sorting, simple
tasks)
Solution prototypes
50+ 10 daysQuick concept proof, Effective
in IA
Limited testable
prototype
Online surveySpecific
questions100+ 10 days
Statistical Significant
Questionnaire design
Mouse Tracking HeatMap
Launched product
200+ 3 days Visual, intuitiveSingle page
analysis
A/B testLaunched product
200++ 15 daysAccurate, Statistical Significant
May need more time for
results
User Interviews ： F2F, Phone, online
User researchers and designers (sometimes product managers) go to interview users together, to enrich shared understanding of our users.
Interviewees: China Suppliers International Buyers
Methods: Factory visits, tradeshows Phone interview, conference calls Online focus group
Member: Mr. ZhouCompany: Zhejiang Dale Garments Co., Ltd.
Jane F. IvanovPresident, Eve Alexander Maternity/Nursing LingerieLocation: USA Industry: Apparel Member since 2005
How to interview (business) users
Aim to achieve rapport (in sync moment) in conversation Let user start telling stories, (where the richest insights lie) Active listening and humility is the most effective way to build
trust and rapport Make user the expert and you are the novice, (vice-versa)
Ask the participants to play the teacher role help to articulate the details
Sometimes you need to show some areas of expertise to make them want to talk to you
Near the end of the interview is a great opportunity to ask more audacious questions (about their business)
Check your own worldview at the door, ready to embrace user’s mental model （ Empathy ）
Don’t fixate on what you expect to learn, cultivate your own general curiosity (about roles in business decisions)
*Interviewing Users: How to Uncover Compelling Insights (Portigal, Steve)
Cross-cultural user research
沉默是金： Silence is Gold Using silence to elicit participants to talk is an art in interview Three kinds of silences in Japanese conversations*
Setting-the-stage silence: Indicate readiness for a shared experience Show mutual respect and humility Interviewer need to match with natural responses and thought pace
Effort silence: Show efforts in thinking and a desire to contribute Interviewer should allow room to think, use silence to encourage the
efforts Failure silence:
Indicate confusion or resistance Interviewer should be attentive to this silence and step in to help
*Interviewing Users: How to Uncover Compelling Insights (Portigal, Steve)
Cross-cultural user research
Think aloud in user testing• Definition: In a thinking aloud test, you ask test participants to use the system while
continuously thinking out loud — that is, simply verbalizing their thoughts as they move through the user interface.
• In Asia, it is impolite to tell someone they have a bad design. • It is embarrassing to admit that you cannot find something. May consider “Retrospective Think-aloud protocol”** after task
completion East Asians need more encouragement, be attentive to silence Sometimes, young/inexperienced facilitator may be able to
collect more negative feedbacks “Teach me how to do it”, asking user to play the teacher’s role
* Thinking Aloud: The #1 Usability Toolhttp://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/** Kuusela, H., & Paul, P. (2000). "A comparison of concurrent and retrospective verbal protocol analysis". American Journal of Psychology (University of Illinois Press) 113 (3): 387–404.
Cross-cultural user research East: Interdependence,
Relationships, Panoramic West: Independent,
Individualistic, zoom in
“Westerners focus on a main or dominant foreground object, while people from East Asia pay more attention to context and background. (East Asian people who grow up in the West show the Western pattern.)”
“When shown complex, busy scenes, Asian-Americans and non-Asian-Americans recruited different brain regions. The Asians showed more activity in areas that process figure-ground relations—holistic context—while the Americans showed more activity in regions that recognize objects.”
How we see it: Culturally different eye movement patterns over visual scenes, Julie E. Boland, Hannah Faye Chua, & Richard E. Nisbett
Differences Between East and West Discovered in People’s Brain Activity Carey Goldberg ， http://tech.mit.edu/V128/N9/culture.html Sharon Begley: West Brain, East Brain
May explain this!
Poor man’s eye tracking HeatmapBased on: http://sourceforge.net/projects/clickheat/
User mouse click tracking and in-page web analysis tools The Problem:
Designers think best in pictures, they need better visual way to see users’ interaction on our page elements
Eye tracking is too expensive in $ and time
Background ： Given a high intent/opportunity to click,
the mouse move and click are much more likely to be closely related to the eye movement*
* Summary of research on eye - mouse syn available from: Edmonds, A., White, R., Morris, D., Drucker, S. Instrumenting the Dynamic Web. Journal of Web Engineering (JWE), Vol. 6, No. 3 (2007), 243-260.
Solution:• Embed click tracking JavaScript to collect user interactions on page.• Real-time adjustment of sampling ratio and/or targeting conditions, i.e.
certain types of users, etc.
Clickheat(flat log file)
Filter by user types
Marmot* @ Taobao, Baidu
(hadoop)
The evolution of Clickheat
*http://www.slideshare.net/taobaoued/marmot
Mouse movement tracking SMT video replay
Moderated remote user testing
Remote user testing tools and methods （ Example test: main user flow ）
Recruiting participants
Participants registration and screening
Inviting participant to testing
Participant accept testing
Record and review testing sessions
Live Recruiting
Ads*
Online Registration
Forms
Phone call or IM
TechSmith UserVUE
UserVUEMorae
Manager
Step 1 ： Live Recruiting
Kiva.org Vs. Amazon gift certificates
Note ： Used Ethnio Recruiting tool first, developed our own tool later.Basic Function ： Pop-in Live Recruiting Ads. Real time control On/Off, Pop-in ratio, etc.Advanced Function ： Conditioning on member types, reg. date, last visit time, etc.
Real time control On/Off, Ratio,
Conditions
Step 2 ： Participants Registration and Screening
Simple questionnaire for making appointment and screening
Live Recruiting support
Basic Function ： Used Open Source LimeSurvey system for Registration form and surveyAdvanced ： Embed chat for answering questions and live recruiting support
Step 3 ： Invite participants to testing
Limited by facilitator resources, tests conducted one by one
Basic Function ： LimeSurvey responses analysis for screening participants
Step 4 ： Participants setup screenTechSmith UserVue
Participant’s screen in testing session
TechSmith UserVueTasks and questions communicated via telephone or IM
Moderator’s Screen
Remote User Testing Analysis ： Morae Manager
Remote Vs. Traditional User Testing
Pros: Testing on Real users (incl. 1st time visitors) Testing on users’ own computer Interactive feedback and communications on major process User Screen Recording and live broadcasting to Observers Can do additional user background interview, possible to do
longitude follow-up studies No travel, low cost and low equipment requirements. Minimum requirement for users.
Cons: No facial or body movement video if no webcam on user side. Possible interruption on user side, may be limited by Internet
speed. Downloading and running application on user’s computer can
be a problem. http://www.boltpeters.com/articles/versus.htmlhttp://www.techsmith.com/uservue/whyremotetest.asp
Remote User Testing Requirements
Need to work with designers and PM to confirm user testing tasks.
Recruiting participants need to be representative to target user group.
From testing plan to pilot testing, close involvement of designer and PM are essential for successful tests.
Alternative remote testing tools tried:
User download/run remote desktop sharing applications Online Conference tools: WebX, GotoMeeting,
TeamViewer, etc. IMs ： QQ remote assistance, MSN etc. VNC: RealVNC, UltraVNC, etc. RDP ： Windows Remote Desktop
No client applications, Remote desktop in Browser Guacamole: HTML5 remote desktop Flash-based
Cloud-based Virtual Machine
Automated simple tasks
Task 2 of 4: Where would you go to sign up for email notifications of the latest products?Task 3 of 4: How to find out the supplier's contact information?Task 4 of 4: How to check out more information about a product you are interested in? Make interaction
prototype testable online:•Used OptimalWorkshop Chalkmark for simple IA tests;•Developed single page interaction prototype tests based on limesurvey and Clickheat mouse tracking
Online card sortingSolving IA problems
Understand a different point of view.
Using OptimalWorkshop OptimalSort, for menu design and IA problems.
R shiny web app for card sorting analysis (WIP)
plot(hclust(dist(Data)))
Structured Online Survey: based on Limesurvey
Problem: Many quick small research requested
Solution:Use panel, survey ads, email survey invitations to quickly collect large amount of target users feedbacks.
Many classical marketing research model can provide deep insight for user research, U&A, perception model, etc.
Questionnaire design with analysis in mind is the key to successful online survey.
Be sure, not sorry
Marketing models
Biplot (MCA, MDS, MDPREF)
Marketing Research: Uncovering Competitive Advantages, Warren F. Kuhfeld, SAS Institute Inc.
A nice exit survey example
Voice from users
Problem in collecting user feedbacks Customer satisfaction surveys: limited coverage
and long time lag Online customer service and feedback form:
manual processing low efficiency, no priority No transparency in user feedback handling, low
motivation in participation Many user suggestions are not specific, not
actionable Operation and management need more
valuable voice from customers……
Solution: User feedback systems with product feature/function setting
Amazon has been collecting all kinds of specific user feedback
Feedback on Ads
Feedback on Search
Taobao is collecting specific feedback on search
Feedback on Search Experience
Alibaba.com Key User Flow Feedbacks
Feedback on Search Experience
Feedback on
Product Details
But, the value lies in how we process these feedbacks
Suggestion Box 2.0
User feedbacks with votesService providers ：•Salesforce.com•Uservoice.com•Getsatisfaction.com
User feedback system solution
Main functions and scenarios
1. User post suggestions2. User interacts on priorities
and solutions3. Company participate in
feedback community, provide feedback on feedback
Advantages Facilitate multiple product
lines, configurable feedback types
Integrated user feedback channel
Unified feedback interface and community
Provide long term customer satisfaction metrics
Interaction among users ： Tag and vote on feedbacks Share and discuss solutions
Interaction with company ：
Transparent processing of user feedback
Tracking feedback status on user feedbacks, increased user satisfaction and motivation
Team and process supports Additional problems
Privacy protection: anonymous and closed feedback Spam control, User admin, Voting limits, Scores, etc. Search and suggestion to consolidate duplicates Pre-defined feedback types and product lines
Customer service team handle questions and complains
Support Ticket tracking, integrate with existing BOPS 、 CS support email system
Satisfaction rating on CS provider
Suggestions are handled by designated PM PM handle TOP suggestions of related product lines,
tracking, organizing, replying Top feedbacks
User research provide integrated summary reports
Provide user satisfaction metrics by product lines Follow-up studies on Top user feedbacks
User feedback
Customer service:Reply (if needed)
Question & Complains
User interaction and
CS reply
CS handling of question & complains
Issue support ticket tracking
Suggestions
Users vote on priorities
PM handle TOP suggestions
User track feedback
status
AB-Testing at different levels
•JavaScript to randomize display of elements•Cookie-based randomization (simple implement)•Not suitable for complex tests, affect page loads•Reverse engineering of Website Optimizer metrics
Frontend
•Low level code for random assignments•Multiple template sets•Complete multivariate test solutionsTemplate
•Low level code for random tests•Useful in algorithm test: search ranking, p4p rules, etc.Backend
Simple in-page design
Interaction process
Workflow, algorithm
The Hippos Vs. Jaguar Process
Set Success Measure(Management)
Collect Ideas(From Everyone)Esp. customers
Select(Approver)
Execute(Agile team)Cross-functional
Test & Evaluation(Iterate)
http://www.slideshare.net/jwaxman74/jared-waxmans-ab-testing-10-principles-to-organize-for-rapid-testing (Adapted from)
Traps:•Management not aligned•No resources – fully booked
Traps:•Ownership issues•Technically impossible
Traps:•Endless debate
Traps:•Perfection
Traps:•Opps…forget about it
Highest Paid Person’s Opinion (HiPPO)
Joint (JIT) Agile Users Aggregated Responses (JAgUAR)
AB-Testing
User Feedbacks
•User feedbacks •Systems approach, standardized process
•User input for priority•Feedback on feedback
Understand real users need
•User research •Validate design ASAP•Avoid HiPPO’s design•User-Centered Interface/interaction
Design effective solutions •Agile Development
•Cross-functional small teams
•Bottom-up communication
•Fast iterations
Fast iterative product
development
•Evaluating performance•Fast, actionable testing results
•Multivariate tests•Clear and concise metrics
Metric-based validation
Collaborative teams and integrated process
User ResearchCustomer service
User ResearchProduct Management
User ResearchDesign and Development
Kaizen
Continuous Improvement
改善http://en.wikipedia.org/wiki/Kaizen
Kaizen (Japanese for "improvement" or "change for the better") refers to a philosophy or practices that focus upon continuous improvement of processes in manufacturing, engineering, supporting business processes, and management. It has been applied in healthcare, government, banking, and many other industries. When used in the business sense and applied to the workplace, kaizen refers to activities that continually improve all functions, and involves all employees from the CEO to the assembly line workers.