exploring the ux research toolbox
DESCRIPTION
This presentation explores the intersection between UX strategy and research: Part 1: Why do research, anyway? Part 2: Understand the landscape Part 3: Pushback & pitfalls Part 4: Exploring the toolbox Part 5: Case Study: ATB Originally presented at VanUE on April 29, 2014.TRANSCRIPT
Exploring the research toolbox: what to use, when and why.
Calgary - Vancouver - Toronto
Anthony HempellDirector, User Experience
Tara Franz,Director, Research
Part 1: Why do research, anyway?Part 2: Understand the landscapePart 3: Pushback & pitfallsPart 4: Exploring the toolboxPart 5: Case Study: ATB
Part 1:
Q: Why do research, anyway?
Part 1:
Q: Why do research, anyway?
“If we knew what it was we were doing, it would not be called research, would it?” -Albert Einstein
Part 1:
Q: Why do research, anyway?
“If we knew what it was we were doing, it would not be called research, would it?” -Albert Einstein
“I believe in innovation and that the way you get innovation is you fund research and you learn the basic facts” - Bill Gates
Why Research?
Why Research?
measurements, facts, observations
“know nothing”words, numbers, symbols
Why Research?
descriptions, correlation, comparison, relationships
“know what”
meaning & purpose
Why Research?
framed experiences, values, context, expertise, intuition;
organizational routines, processes, practices and norms
understanding & capability
“know how”
Why Research?confidence to act
judgement, ethical and aesthetic values
“know why”
So how do we do that?
gather server logs, conduct surveys, perform usability tests
comparison & analysis of data
communicate and evangelize findings
turn insights into policies, methods and actions
Part 1: A: We want to fill our gaps in the data to create useful information, knowledge & wisdom which can be used to make the world a better place.
Part 2: Understand the landscape
Part 2: Understand the landscape
“If you know the enemy and know yourself, you need not fear the result of a hundred battles.” - Sun Tzu
Some practical questions before we
begin...What do we already know?
Who are the users? Who are the stakeholders?
Where are they located?What are we trying to discover?When do we need our findings?
How are the findings going to be used?
What do we already know?
What research results do we already have access to?
How recent is it?
How was it collected?
Is it valid?
What does it tell us?
Has it been converted into organizational assets like policies, procedures, shared knowledge?
Who are the users?
What research has been done on current audience?
Is any of it qualitative?
How has the current audience / user / customer base been determined?
Are there demographic aspects to customer base that are of interest -- language, age, mobility, cultural, technology use, etc.
Who are the stakeholders?
Who has ultimate accountability (financial and otherwise)?
Who is the primary point of contact / responsibility for coordination?
Who else needs to be consulted? What are their roles?
Who needs to be informed and when?
Who would be most affected if the project goes poorly?
What are the desired outcomes + effects?
Where users + stakeholders located?
Local? Provincial?
National? International?
Urban / Suburban / Rural
Regional attitudes / behaviours?
When do we need our findings?
Deadlines -- what is driving them?
Possibility of future research?
How are our findings going to be used?
Creating requirements?
Concept creation or validation?
Setting policies / product direction?
Within small project team or communicating across departments?
Part 2: Gather as much data about your surroundings as possible.
Part 3: Pushback & Pitfalls
Part 3: Pushback & Pitfalls
“You can go anywhere you want if you look serious and carry a clipboard” - Murphy’s Law
Part 3: Pushback & Pitfalls
“You can go anywhere you want if you look serious and carry a clipboard” - Murphy’s Law
“We don’t devote enough scientific research to finding a cure for jerks” - Bill Watterson
Some common pitfalls“We already have a research department
that does that”
“Marketing did some market research last quarter, we can use that”
“Let’s put a survey up on our website”
“That sounds like it will take too long / be too expensive -- let’s talk to some of my
friends / co-workers / kids instead”
“Let’s get the users to tell us what they want” (a.k.a “Rainbow Unicorn”)
Some common pitfalls“We already have a research department
that does that”
“Marketing did some market research last quarter, we can use that”
“Let’s put a survey up on our website”
“That sounds like it will take too long / be too expensive -- let’s talk to some of my
friends / co-workers / kids instead”
“Let’s get the users to tell us what they want” (a.k.a “Rainbow Unicorn”)
Some common pitfalls - why they happen
Organizational pressure for concrete results, yesterday
Research not seen as valuable / waste of time
Lack of knowledge about research methods
Lack of understanding of research benefits
Quote
Research
Strategy
Design
Culture
Testing
Novice Beginner Intermediate Advanced Enlightened
NoneAd-hoc,
AnecdotalData-driven
Qualitative + quantitative
Triangulation: data & observation;
passive & active.
Not used
Ad-hoc
Never
No knowledge
Reactive; HIPPO
Creative Chaos
Rarely
Awareness of value
Focus on business requirements
"Best practices"
Occasionally
Individual champions;
traction dif!cult
Balance needs of business & users
Style guides + processes de!ned; ideation & creativity
Sometimes
Some managerial advocates +
success
Business is customer-driven
Agile / Lean: small teams, rapid design sessions, iterative
Always
Empowered by C-level executive
Adapted from: Shane Morrishttp://blogs.msdn.com/b/shanemo/archive/2006/12/18/user-experience-maturity-model-microsoft-style.aspx
"We don't have time / money for
that!""Just do it."
"Think before you act."
"Use lessons learned to predict, plan and create opportunity."
"Review, learn, adjust, execute, review, learn…"
Jakob Nielsenhttp://www.nngroup.com/articles/usability-maturity-stages-1-4/
0 1 2 3 4
UX Maturity Model
Quote
Research
Strategy
Design
Culture
Testing
Novice Beginner Intermediate Advanced Enlightened
NoneAd-hoc,
AnecdotalData-driven
Qualitative + quantitative
Triangulation: data & observation;
passive & active.
Not used
Ad-hoc
Never
No knowledge
Reactive; HIPPO
Creative Chaos
Rarely
Awareness of value
Focus on business requirements
"Best practices"
Occasionally
Individual champions;
traction dif!cult
Balance needs of business & users
Style guides + processes de!ned; ideation & creativity
Sometimes
Some managerial advocates +
success
Business is customer-driven
Agile / Lean: small teams, rapid design sessions, iterative
Always
Empowered by C-level executive
Adapted from: Shane Morrishttp://blogs.msdn.com/b/shanemo/archive/2006/12/18/user-experience-maturity-model-microsoft-style.aspx
"We don't have time / money for
that!""Just do it."
"Think before you act."
"Use lessons learned to predict, plan and create opportunity."
"Review, learn, adjust, execute, review, learn…"
Jakob Nielsenhttp://www.nngroup.com/articles/usability-maturity-stages-1-4/
0 1 2 3 4
Quote
Research
Strategy
Design
Culture
Testing
Novice Beginner Intermediate Advanced Enlightened
NoneAd-hoc,
AnecdotalData-driven
Qualitative + quantitative
Triangulation: data & observation;
passive & active.
Not used
Ad-hoc
Never
No knowledge
Reactive; HIPPO
Creative Chaos
Rarely
Awareness of value
Focus on business requirements
"Best practices"
Occasionally
Individual champions;
traction dif!cult
Balance needs of business & users
Style guides + processes de!ned; ideation & creativity
Sometimes
Some managerial advocates +
success
Business is customer-driven
Agile / Lean: small teams, rapid design sessions, iterative
Always
Empowered by C-level executive
Adapted from: Shane Morrishttp://blogs.msdn.com/b/shanemo/archive/2006/12/18/user-experience-maturity-model-microsoft-style.aspx
"We don't have time / money for
that!""Just do it."
"Think before you act."
"Use lessons learned to predict, plan and create opportunity."
"Review, learn, adjust, execute, review, learn…"
Jakob Nielsenhttp://www.nngroup.com/articles/usability-maturity-stages-1-4/
0 1 2 3 4
Company A
Company B
Company C
Company D
Some common pitfalls - how to avoid
Know your research question. Always have concrete, measurable goals for your research that you can refer to.
Build a research strategy and plan.
Be willing to listen to alternative points of view, but don’t deviate from sound research practices.
Protect the validity of your methods. It is your basic currency.
If you’re getting pushback, take baby steps.
Focus on the value.
Don’t force it. If the glass slipper doesn’t fit, it doesn’t fit.
Part 3: Observe and understand the organization you’re working with. Propose measured improvements. Make friends.
Part 4: Exploring the toolbox
“If all you have is a hammer, everything looks like a nail.” - Abraham Maslow
Focus groupsIn-depth interviews (in person / remote)EthnographyCard sortingUsability testingUsability benchmarkingSurveysObservationMessage board miningCoding customer feedback emailsA/B testingWeb AnalyticsDiariesEye trackingOnline Panels / Communitiesand more...
http://nform.com/cards/
Let’s look in the toolbox
Research dimensionsBasic vs AppliedCross-sectional vs LongitudinalResearch purpose: Exploratory, descriptive, explanatory
Data collection techniques:Quantitative vs QualitativeSmall vs Large sampleModerated vs UnmoderatedAttitudinal vs Behavioural
http://qualitative.wikidot.com/dimensions-of-research
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Moderated• Can observe subtle cues:
non-verbal behaviour, tone of voice, etc.
• More chances for further inquiry based on context
• Harder to eliminate biases• Much higher quality of data
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Unmoderated• Can facilitate much larger sample
sizes• Not as many constraints with
different 9me zones• Limited context• Media biases• Respondents have more 9me to
par9cipate (send photos, diaries, ponder thoughts)
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
• Small test group but deeper understanding• Gathering informa9on or themes from texts, conversa9ons or
interviews• Open ended; changeable. There is a maybe. • Opportunity for flexibility; serendipity
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
• More closed questioning with directed response (pick Yes or No, there is no maybe)
• Gathering data with an “instrument”• Derive measures or variables (operationalization)• Error of measurement
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
• Hybrid: gather data qualita9vely and then code into variables to make inferences quan9ta9vely.
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Discovering mo9va9ons in context
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix BLUE = measures aUtudes
RED= measures behaviours
Focus groups
In person interviews
Ethnography
Usability tes9ng
Telephone interviews
Card sor9ng
Diaries
Online panels
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Observing and tracking behaviours in context
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Observa9on
Usability benchmarking
Eye tracking
BLUE = measures aUtudes
RED= measures behaviours
Online panels
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Discovering general themes and aUtudes
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Message board mining
Customer feedback emails
BLUE = measures aUtudes
RED= measures behaviours
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Gathering data about behaviours
Mod
erated: high eff
ort / high context U
nmoderated: low
er effort / lower context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Web Analy9cs
A/B tes9ng
Customer feedback emails
Unmoderated usability tes9ng
BLUE = measures aUtudes
RED= measures behaviours
Mod
erated: high eff
ort / high context
Unm
oderated: lower effort / low
er context
Quan8ta8ve: how many and how much
Qualita8ve: why and how to fix
Focus groups
In person interviews
Ethnography
Surveys
Web Analy9cs
Observa9on
Usability benchmarking
Usability tes9ng
A/B tes9ng
Message board mining
Customer feedback emails
Telephone interviews
Card sor9ng
Unmoderated usability tes9ng
Eye tracking
BLUE = measures aUtudes
RED= measures behaviours
Diaries
Digital Ethnography
Online panels
Opportunities:Triangulation
What data do you currently have? How was it collected? Consider using a method that is complimentary.
Do you have the opportunity to run two or more research methods? Use different methods to look at the same problem from different angles.
Part 4: Know the strengths and weaknesses of every method and pick wisely. If possible, research with more than one.
Part 5: Case Study: ATB Financial
“Our youth now love luxury. They have bad manners, contempt for authority; they show disrespect for their elders and love chatter in place of exercise” - Socrates
• Problem: –Required insight and
understanding of the millennial demographic to increase market share within the group
–Make something
awesome
ATB Case Study
• what do we already know? – Lots of data about ATB customer needs and
current use –Also were aware of Millennial research and
some overarching themes
ATB Case Study: Landscape
• Who are the users? –Target of urban Millennials and drafted up a
demographic framework to contain the sampling
• Who are the stakeholders? –Emerge group at ATB and big wigs –Ensure we had a plan to have them involved
ATB Case Study: Landscape
• Where are they located?–Calgary, Alberta
• What are we trying to discover? –Something we can take to market to engage a new
market for ATB
ATB Case Study: Landscape
• When do we need our findings? –We had 3 months time
• How are the findings going to be used? – To create a new offering
ATB Case Study: Landscape
• The plan:- Secondary research review on Millennials and ATB data - Ethnographic Immersion – be one with the animals
- DEBRIEF (this is where the real insight starts to happen and stakeholders are involved) - User Interviews – understand the ways and the why
- DEBRIEF again – what have we learned? What else do we want to know? Let’s get our persona on!
- Naïve Expert and Extreme Customer Interviews (Kingdon; Science of Serendipity. 2012)
- DEBRIEF – what have we learned about solving problems?- Innovation session - The big write up and vote - In field feedback (this is when you call your friends)
- RDL and prototyping- Concept testing - Finalize and forge ahead!
ATB Case Study: Toolbox
- How was everyone kept in the loop?
Collaboration
Remember the triangle?
- Secondary research review on Millennials and ATB data
- Ethnographic Immersion – be one with the animals - DEBRIEF (this is where the real insight starts to happen and
stakeholders are involved)
Stage One: Foundation
- User Interviews – understand the ways and the why - DEBRIEF again – what have we learned? What else do we
want to know? Let’s get our persona on!
• Semi structured phone and in person ethnographic interviews: we dined, we coffee-d, we chatted, we made life plans together
• Debrief goal: Personas
Stage Two: Patterns
- Naïve Expert and Extreme Customer Interviews (Kingdon; Science of Serendipity. 2012)- DEBRIEF – what have we learned about solving problems?- Innovation session - The big write up and vote
Stage Three: Innovation
• Explored creative ways to solve the problems we had: personal organizers, teachers, support workers, rehabilitation counselors
• Spoke with people who were on the extreme side of everything we discovered: wealthy and homeless
• Debrief goal: 5 solid ideas with a framework. • Opportunity to vote on favorite idea • In field feedback
- RDL: include your stakeholders and sometimes users - Prototyping- Concept testing:
- recruit the right people - Test on the right screen
- Finalize and forge ahead!
Stage Four: Building
Anthony [email protected]
Web: www.uxguys.comTwitter: @uxguys
Thank You!
Tara [email protected]
Web: about.me/tarafranzTwitter: @tarafranz