research to inform development decisions, usertesting everyday insights roadshow, august 23, 2016
TRANSCRIPT
![Page 1: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/1.jpg)
Research to Inform Development DecisionsSteve Fadden, Ph.D.
Director, Analytics UX Research, Salesforce
Lecturer, Professional Faculty, UC Berkeley School of Information
Presented at the UserTesting Everyday Insights Roadshow, San Francisco, CA
August 23, 2016
Image: https://pixabay.com/en/photos/spiral%20staircase/
![Page 2: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/2.jpg)
Image: https://www.flickr.com/photos/jiuguangw/5134925627
Agenda1. Introductions
2. Process
3. Formative Methods
4. Case study
![Page 3: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/3.jpg)
About me
Steve Fadden
@sfadden
linkedin.com/in/stevefadden
![Page 4: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/4.jpg)
About youFocus
● Research
● Design
● Engineering
● Marketing
● Management
● Other?
Research experience
● Over 5 years
● 2-5 years
● Up to 2 years
● None
![Page 5: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/5.jpg)
About the research processGenerative: Creation
Formative: Evaluation
Summative: Assessment
Images: https://commons.wikimedia.org/wiki/File:Idea.png,
https://pixabay.com/en/selection-pick-chose-choice-select-443127/,
https://pixabay.com/en/graph-statistics-chart-infochart-29709/
![Page 6: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/6.jpg)
About the research processGenerative: Creation
Formative: Evaluation
Summative: Assessment
Images: https://commons.wikimedia.org/wiki/File:Idea.png,
https://pixabay.com/en/selection-pick-chose-choice-select-443127/,
https://pixabay.com/en/graph-statistics-chart-infochart-29709/
![Page 7: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/7.jpg)
Formative methods inform progressFormative
During development
Lower fidelity
Focused on improvement
Aid decision-making
Faster & lighter
Summative
Ready to launch
High fidelity
Focused on measurement
Understand current state
More comprehensive
![Page 8: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/8.jpg)
Technique 1: Understand the problem
![Page 9: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/9.jpg)
Image: https://pixabay.com/en/detective-magnifying-glass-viewing-788592
Gain insights:
● Evidence of problems
● Opportunities to solve
Based on:
● Recent events
● Specific details
● Feelings and perceptions
● Future behaviors, responses
Critical incident questions
![Page 10: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/10.jpg)
Process1. Confirm user profile
2. Identify time since last experience
3. Gather details
a. Description
b. Actions taken
c. Feelings
d. Outcome
e. Future actions and responses
Reference: http://www.usabilitynet.org/tools/criticalincidents.htm
![Page 11: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/11.jpg)
Applying your findings1. Validate problems
2. Identify opportunities
3. Understand expectations
4. Identify scenarios
5. Build empathy
Image: http://www.freestockphotos.biz/stockphoto/17339
“I needed to share a PDF with a friend, and we use Dropbox, but I hadn’t used it in a while. I logged in through my browser, dragged the PDF to Files, and then saw the PDF open. I expected Dropbox to start uploading. I hit back, created a folder in Dropbox, uploaded the PDF to it, clicked share to add my friend, and sent her the link.”
![Page 12: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/12.jpg)
Technique 2: Assess expectations
![Page 13: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/13.jpg)
Expectation testsGain insights:
● Mental model
● Categorization, labels
● Confusion areas
With:
● Nonworking interface
● No text/labels
● Basic task information
Image: https://pixabay.com/en/photos/man%20thinking/
![Page 14: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/14.jpg)
“Greeking” technique1. Identify important tasks
(e.g. frequent, critical)
2. Write scenario
3. Write “first step”
questions
4. Gather feedback
5. Ask for group names
Example
Scenario: “You are exploring colleges, and need to make sure the courses look interesting and financial aid is available.”
Instruction: “How would you expect to start to apply for financial aid?”
Reference: Thomas S. Tullis, 1998. A method for evaluating web page design concepts. CHI 98 conference summary on human factors in computing systems
![Page 15: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/15.jpg)
“Greeking” activityScenario: You are exploring colleges,
and need to make sure the courses look
interesting and financial aid is available.
How would you expect to:
1. Find out more about financial aid?
2. View information about courses
offered?
3. Contact the department in charge
of Athletics?
![Page 16: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/16.jpg)
“Greeking” resultsHow would you expect to:
1. Find out more about financial aid?
2. View information about courses
offered?
3. Contact the department in charge
of Athletics?
What would you call each group?
12
3
“Academics” “Administrative”
“Connections”
![Page 17: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/17.jpg)
Technique 3: Prioritize needs
![Page 18: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/18.jpg)
Kano model
Absent Present
Dissatisfaction
Satisfaction
Adapted from: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results
![Page 19: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/19.jpg)
Kano model
Absent Present
Dissatisfaction
Satisfaction
Adapted from: http://uxmag.com/articles/leveraging-the-kano-model-for-optimal-results
Exciting
(truly unexpected)
Perform
ance
(m
ore is better) Basic
(required)
![Page 20: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/20.jpg)
Kano model (automobile example)
Absent Present
Dissatisfaction
Satisfaction
Seat-warmers
Bluetooth
Intuitive GPS
H
orsepow
er
Fuel econom
y
Airbags
Adjustable seats
![Page 21: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/21.jpg)
Kano question processProcess
● Assemble list of features
● Pose realistic scenarios for context
Assess for each:
● Importance of feature
● Satisfaction if available
● Satisfaction if not available
Scenario: “Imagine you are using our mobile app to find a restaurant.”
Question 1: “How would you feel if the app allowed you to filter results by distance?”
Question 2: “How would you feel if the app did not allow you to filter results by distance?”
Question 3: “How important is this function to you?”
![Page 22: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/22.jpg)
Kano question results
If present :( If present :)
If absent :)
If absent :(
Map answers to “product
allowed you to...”
questions
Map answers to
“product did not allow
you to...” questions
![Page 23: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/23.jpg)
Kano question results
If present :( If present :)
If absent :)
If absent :(
Providing
this feature is
dissatisfying
Providing
this feature
is satisfying
Not providing this feature is satisfying
Not providing this feature is dissatisfying
![Page 24: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/24.jpg)
Kano question results pattern
If present :( If present :)
If absent :)
If absent :(
Providing
this feature is
dissatisfying
Providing
this feature
is satisfying
Not providing this feature is satisfying
Not providing this feature is dissatisfying
![Page 25: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/25.jpg)
Technique 4: Evaluate the solution
![Page 26: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/26.jpg)
Usability studyGoals
● Explore concepts
● Assess effectiveness
● Compare competitors
● Validate performance
Measures
● Learnability
● Efficiency
● Memorability
● Errors
● Satisfaction
Process
1. Plan (goals, participant criteria)
2. Prepare (recruit, test materials)
3. Conduct (active observers)
4. Analyze (issues, insights)
5. Report (fixes, future opportunities)
Reference: https://www.nngroup.com/articles/usability-101-introduction-to-usability/
![Page 27: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/27.jpg)
Usability studyRITE - Rapid Iterative
Testing & Evaluation
(address problems)
Discount (find problems)
Traditional (benchmark
performance)
Aim to:
● Test representative users● Follow script with actual tasks● Use high-fidelity interface
Key elements:
● Observation by key decision makers (PM, design, development)
● Agree on critical problems and make changes to interface after each session
References: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests (2nd Ed.), Rubin, J. & Chisnell, D. (2008)
Jakob Nielsen, 1989. Usability engineering at a discount, Proceedings of the third international conference on human-computer interaction on Designing and using
human-computer interfaces and knowledge based systems (2nd ed.); Image: The RITE Way to Prototype, J. Shirey et al., UX Magazine, 2013
Using the RITE method to improve products; a definition and a case study, M. Medlock et al., Presented at UPA, 2002.
![Page 28: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/28.jpg)
Example: Understanding data export
![Page 29: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/29.jpg)
Approach used● Background
○ Urgently-needed integration with
a 3rd party product
○ Very visible launch event was
already scheduled
○ Small team, no dedicated research
support
○ “Boil the ocean” effort: Was
concept and functionality
understood, desired, and usable?
Process
Critical incident
Expectation
Prioritization
Usability evaluation
![Page 30: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/30.jpg)
Critical incident elicited rich feedback“Consider the last time you had to export data. Describe why you
needed to export data, and list the steps you remember from that
process. (If you haven’t exported data before, or don’t remember
the last time, just skip to the next question).”
“I’m pretty old school, so I export my credit card transaction data about every quarter. My credit card site has a button to export to CSV, so I just click that and it downloads to my computer.”
“We have our marketing, sales, and inventory data in different systems. I have to export data from each system in order to combine it into a spreadsheet for my stakeholders. The export process is easy. Combining the data is more involved.”
![Page 31: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/31.jpg)
Expectations gathered from storyboard“Consider the concept presented on the next 4 slides. As you
read through the concept, comment on anything you find to be
confusing, problematic, useful, or appealing about the concept.”
1. 2. 3. 4.
100%
![Page 32: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/32.jpg)
Initial expectations highlight concerns1.
2.
3.
4.
100%
“Doing this would require a lot of clicks, even for a small number of columns.”
“You should embed best practices for naming here. Otherwise, the result could be messy.”
“Will we be able to save the mappings? That could save time in the future.”
![Page 33: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/33.jpg)
Directed feedback emphasized value and priorities
“It’s great that you don’t have to jump around different parts of the system to do this. Very valuable to be able to complete this from one place.”
“It’s useful, but only if I can save it and reuse it in the future. Wouldn’t use it for a one-time export.”
“Hi, I wanted to follow up to reiterate that this is a REALLY COOL idea and it fills a much needed requirement for our use of the product. Please consider me for future studies like this, because we need this functionality!”
![Page 34: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/34.jpg)
Usability sessions identified more issues
● RITE with 4 participants over 2 days
● Key changes: Terminology, embedded instructions, icons
● Stakeholder advocacy (and gratitude)
“It says ‘save’ but I’m just modifying the settings. Unless I can re-use this later?”
“I haven’t selected the columns yet! Why can’t I go back?”
![Page 35: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/35.jpg)
Some formative evaluation results● Evidence of problems
● Value of overall solution
● Prioritizations and tradeoffs
● Issues with flow, terminology, design elements
● Importance of incorporating formative evaluation into
process
![Page 36: Research to inform development decisions, UserTesting Everyday Insights Roadshow, August 23, 2016](https://reader031.vdocument.in/reader031/viewer/2022022609/587b4d461a28abff1a8b51cf/html5/thumbnails/36.jpg)
Thanks!
Image: https://en.wikipedia.org/wiki/File:TulipStair_QueensHouse_Greenwich.jpg