user testing: adapt to fit your needs
TRANSCRIPT
Sponsors
User Testing: Adapt to Fit Your Needs#DenStartupWeek #EffectiveUI
Agenda
1. Intro 2. What is evaluative research? 3. How To 4. Engaging the whole team 5. What to do next
Ari WeissmanLead Experience Architect
888.310.5327
• 10+ years global experience
• Responsible for all things UX, from research through experience design
• Project lead for TimeWarner Cable, AMEX, FreemanCo, and Securian
@ EffectiveUI
Lys Maitland Senior Experience Planner
888.310.5327
• 15+ years global experience
• Responsible for all things UX, from research through experience design
• Research lead for Davita, Scottrade, NextGear, Securian, CenturyLink
@ EffectiveUI
Insight-driven digital product company
Locations
2162 Market StreetDenver, CO 80205
DENVER
274 North Goodman St, Unit B264Rochester, NY 14607
ROCHESTER
85 Broad Street, 18th fl.New York, NY 10004
NEW YORK CITY
Our clients
What is evaluative research?
[a.k.a. user testing]
Generative v. Evaluative
Generative research
Helps define the problem.
The goal of generative research is to look to the activities, environments, interactions, objects and people to find opportunities for solutions and innovation. These solutions could be new products or experiences or they could be an update or improvement to an existing one.
Evaluative research Evaluates an existing design (in prototype form or in final form).
The goal of evaluative research is to test the existing solution to see if it meets people’s needs, is easy to access and use, and is enjoyable. This type of research should be conducted throughout the development lifecycle, from early concept design (think rough sketches or prototypes) to the final site, app, or product.
DEFINE DISCOVER DESIGN DEVELOP DEPLOY
EVALUATIVE RESEARCH
Field Research Iterative Concept Testing Usability Testing
Define Problem
Problem Approach
Gather Data
Observational Research
Benchmark Current Site
Distinguish Insights
Analytics Assessment
Create Concepts
Concept Test
Explore Solutions
Build Prototypes
Test & Refine
Collaborate On Options
Discover Gaps & Problems in Experience
Analytics Strategy
Implementation/ Change Management
Continuous Measurement
Test & Refine
Some examples of evaluative research
Testing an IA or design learnabilityCLICK TEST
Testing flows and contentUSABILITY TESTING
Testing and measuring current state for redesign
BENCHMARKING
Closed card sort to validate an IACARD SORTING
Iterative testing of flows and concepts, some generative research
ITERATIVE DESIGN TESTING
Benefits of User Testing
1 Validate assumptions 4 Provide justification for additional effort
2 Decide between options 5 Reveal how users think about the problems your solution solves
3 Discover hidden issues 6 Removes opinion from the design process
How To
How To
1. Define the goals
2. Define the audience
3. What to test
4. $ -> $$$
5. Debrief
6. Analysis
7. Outcomes/Deliverables
What are your goals?
888.310.5327
Some questions to ask: - Why are you testing and what will
you do with the data?
- What do you need to know / learn / validate? Why?
Who is your audience?
888.310.5327
Some questions to ask: - Who will use of the thing you’re
building?
- Are there different groups of users? If so, which have the highest priority?
- Where are they?
Recruiting
Screener • Script used to recruit
• Intro to the study
• Questions to find participants who demonstrate desired behaviors
• Offer an incentive
Schedule 1 hour interviews with at least 30 minutes between sessions
Recruit • 5-7 participants per
segment
• 5 participants finds 85% of usability issues
1 2 3
Participants are good to the extent they represent your target. If your participants don't match your target, your study will be useless. You can learn valuable things by asking the right people the wrong questions. If you are talking to the wrong people it doesn't matter what you ask. Bad participants undermine everything you’re trying to do.
— Erika Hall, author of Just Enough Research
“”
Run the first session as a pilot to work out the kinks and refine the moderators guide. Schedule an hour
after the first to re-tool as needed.
PRO TIP
What can I test?
ANYTHING
- CONCEPT
- HAND SKETCHED MESSY
- HAND SKETCHED NEAT
- WIREFRAMES
- VISUAL DESIGN
- CLICK-THROUGH PROTOTYPE
- CODED PROTOTYPE
- FINAL PRODUCT
The quick and dirty
Why quick and dirty?
- NO TIME
- NO MONEY
- NO FACILITIES
- TESTING SOMETHING SMALL
Quick and dirty
Location Anywhere!
Logistics Find a place
Find some people
Ask some questions
Write down answers
Costs Minimal to none
Fidelity to test Any
1 2 3 4
Middle of the road
Why middle of the road?
- NEED SOME DOCUMENTATION TO
SHARE WITH TEAM OR SAVE
- TOO MUCH CONTENT FOR QUICK
AND DIRTY
- WANT TO HAVE OBSERVERS
- DESIRED USERS ARE NOT READILY
AVAILABLE
Middle of the road
Location Preferably in context,
though may be remote
In a quiet room at a table
Logistics Schedule a location and
time for one hour
Recruit participants and consider incentive
Prep discussion guide
Create data capture sheet
Audio/Video recording
Costs Location
Recruiting and incentives
Screen sharing/Recording tool
Fidelity to test Typically wireframe or
above
1 2 3 4
Formal
Why formal testing?
- REQUIRES FORMAL DOCUMENTATION
- DESIRED USERS ARE NOT READILY
AVAILABLE
- STRICT DISCUSSION GUIDE
REQUIREMENTS
- LOTS OF THINGS TO TEST
- NEED TO INCLUDE NOTETAKERS AND
OBSERVERS
- MULTIPLE DEVICES OR COMPLEX
TECHNOLOGY REQUIREMENTS
Formal
Location Quiet room with a table
Separate location for observers
Quality phone if remote
Logistics Schedule a location and
time for two hours
Recruit participants and distribute incentive
Prep/share/revise discussion guide
Create data capture sheet
Audio/Video recording
Costs Location
Recruiting and incentives
Screen sharing/Recording tool
Snacks and beverages
Transcriptions
Fidelity to test Typically wireframe or
above
1 2 3 4
FacilitationDo you have 3 more hours?
Facilitation: The highlights
Asking questions Ground your questions
• Will you How have you…?
• In general In this specific instance…?
• Tell me how Show me how…?
Ask why? A lot!
Rhythm • Ask
• Listen
• Probe
• Validate
3 Types of tasks • Specific verb based
tasks (do, send, complete)
• Scavenger hunt (find, discover)
• Interview (tell me about)
1 2 3
So what I heard was, you do X and Y because of A
and B, but a little bit of 123 would make it better. Is
that right?
PRO TIP
The Debrief888.310.5327
Debrief after EVERY session
It’s a quick assessment of a session:
- Summary of the participant
- What went well
- What needs improvement
- Insights or surprises
- Opportunities
Analysis and Synthesis
Trends Look for trends across all participants or within a participant segment
Good and Bad Document what doesn't work AND what does work
Prioritize Choose the most important findings to share and focus on solving
Think big picture Is the issue identified a symptom of a bigger problem?
Quotes Identify quotes that capture the essence of your insights
Solutions What solutions were identified by participants or observers?
Future Opportunities What do the outcomes suggest would be a good next step?
Centralize and share All data in one place for access and redundancy. Shit happens.
Examples of analysis
OUTPUTS
Spreadsheet
39
Post-its
40
Live
Examples of evaluative research results
OUTPUTS
$
quick and dirty
$
quick and dirty
$$
middle of the road
$$$$
formal
Engaging the Whole Team
Engage your team at every step
- CHOOSING GOALS AND AUDIENCE
- REVIEWING DISCUSSION GUIDE
- OBSERVING AND NOTE TAKING
- DEBRIEFING
- PROBING DURING ANALYSIS
- PRIORITIZING RESULTS
- DETERMINING NEXT STEPS
What to do with outcomes
What to do with outcomes
1 4
2 5
3 6
Share your findings
(email, present, print, posterize)Create a roadmap with priorities
Identify opportunities Support ideation
Stay engaged!
Be available to answer questions.Make recommendations
7 Test Again!
Questions?
Vragen Preguntas 問題 Des questions שאלותвопросов คำถาม frågor
Thank you.Ari + Lys
P.S. We’re hiring! http://www.effectiveui.com/company/careers/
www.linkedin.com/company/effectiveuiLINKEDIN
https://twitter.com/effectiveui TWITTER
www.facebook.com/InsideEffectiveUIFACEBOOK
www.instagram.com/effectiveuiINSTAGRAM
https://medium.com/@effectiveui MEDIUM
Stay connected with EffectiveUI
https://abookapart.com/products/just-enough-researchJUST ENOUGH RESEARCH BY ERIKA HALL
http://rosenfeldmedia.com/books/interviewing-users INTERVIEWING USERS BY STEVE PORTIGAL
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users WHY YOU ONLY NEED TO TEST WITH 5 USERS BY JAKOB NIELSEN
Mentioned References