nytech "measuring your user experience design"

11

User Experience Group

Measuring Your User Experience DesignA dozen methods for ensuring user acceptance and business success

Wednesday February 29, 2012, 6:00 PM

New York Institute of Technology

16 W. 61st St., 11th Floor AuditoriumNew York, NY 10023

Organized by: Charles L Mauro CHFPPresident/Founder

MauroNewMedia

23 East 73rd. Street Suite 5F

New York, NY 10021

V 212.249 3683 Ext. 101

[email protected]

Skype mauro.charles

Certified Human Factors Engineering Professional

Copyright MauroNewMedia 2012, All Rights Reserved

2

Business Objective For Measuring Your User Experience?

Earliest identification of the most critical problems/opportunities at the least cost.


3

Rob Tannen Ph.D. – [email protected]

► Rob Tannen, PhD, Director of Research, Bresslergroup - Rob Tannen is an expert in designing products, interfaces, and systems that accommodate the complexities of human behavior and capabilities. He has researched cockpit interfaces for U.S. Air Force, designed trading floor order systems for the New York Stock Exchange, and created touch screen applications for consumer appliances. Rob is Director of Research and Interface Design at the product development firm Bresslergroup. He has a PhD in human factors and is a Certified Professional Ergonomist. He blogs on design and research trends for FastCompany.com.


http://www.bresslergroup.com/

http://www.bresslergroup.com/

4

Researching the Researchers


5

Types of Research Methods

Quantitative

Qualitative

Generative Evaulative

12 fMRI Brain Imaging

10 Eye Tracking

6 Lab-Based Testing

8 Professional Heuristics

1 Contextual Observation

2 Remote Ethnography

11 Online UX Concept Surveys

9 Online Card Sorting

7 Large Sample On-Line Behavior Testing

5 Ergonomic Observation

4 Focus Groups


6

Four Categories of User Research Methods

► Document

► Measure

► Extension

► Enhancement


7

Methodology: Contextual Observation/Ethnography

► Business problem► How are people actually using products versus how they were designed?

► Description► In-depth, in-person observation of tasks & activities at work or home.

Observations are recorded.► Benefits

► Access to the full dimensions of the user experience (e.g. information flow, physical environment, social interactions, interruptions, etc)

► Limitations► Time-consuming research; travel involved, Smaller sample size does not

provide statistical significance, Data analysis can be time consuming► Data

► Patterns of observed behavior and verbatims based on participant response, transcripts and video recordings

► Tools ► LiveScribe (for combining audio recording with note-taking)

Cost / respondent: Low – Moderate – High

Statistical validity: None – Some – Extensive

1


8

Methodology: Contextual Observation/Ethnography1


9

Methodology: Remote Ethnography

► Business problem► How are people actually using in their environment in real-time?

► Description► Participants self-record activities over days or weeks with pocket video

cameras or mobile devices, based on tasks provided by researcher.► Benefits

► Allows participants to capture activities as they happen and where they happen (away from computer), without the presence of observers. Useful for longitudinal research & geographically spread participants.

► Limitations► Dependence on participant ability to articulate and record activities,

Relatively high data analysis to small sample size ratio► Data

► Patterns based on participant response, transcripts and video recordings► Tools

► Qualvu.com



2


10

Methodology: Qualitative Data Analysis

► Business problem► What is the meaning of subjective/unstructured data and feedback provided

by users?► Description

► Tools for capturing and reviewing unstructured data such as interviews, feedback and observed behaviors.

► Benefits► Provides framework for inputting and organizing data during the research

activities and then to review and visualize following data gathering► Limitations

► Does not define analysis method; only provides framework; Dependent on quality and content of metadata inputs and tagging

► Data► Tagging or grouping of text/content information into meaningful categories

► Tools ► ManyEyes.com, ReFramer.com,



3


11

Methodology: Focus Groups

► Business problem► What are perceptions and ideas around products/concepts?

► Description► Moderated discussion group to gain concept/product feedback and inputs;

can include screens, physical models and other artifacts► Benefits

► Efficient method for understanding end-user preferences and for getting early feedback on concepts , particularly for physical or complex products that benefit from hands-on exposure and explanation

► Limitations► Lacks realistic context of use; Influence of participants on each other

► Data► Combination of qualitative observations (like ethnographic research) with

quantitative data (e.g. ratings, surveys)► Tools

► See qualitative data analysis



4


12

Methodology: Observational Ergonomics

► Business problem► What are the physical implications of a product design?

► Description► Technique for identifying and documenting physical usability issues that

can include product, gestural and touch screen interactions► Benefits

► Technique for identifying and documenting physical usability issues that can include product, gestural and touch screen interactions

► Limitations► Technique for identifying and documenting physical usability issues that

can include product, gestural and touch screen interactions► Data

► Ergonomic issues list (frequency, priority); Visualization of ergonomic issues on images or videos

► Tools ► RCPS (Reach/Clearance/Posture/Strength) Protocol, Comfort

Questionnaire, Measurement Tools (TPS)Cost / respondent: Low – Moderate – High


5


13

Methodology: Observational Ergonomics5


14

Methodology: Observational Ergonomics5


15

Universal Methods of Design


16

Charles L Mauro CHFP ([email protected])

► Charles L Mauro, CHFP is President of MauroNewMedia, a New York-based consulting firm specializing in usability engineering and user interface design, founded in 1975. Over his 35 year career, Charles has been retained by many leading corporations and startups covering a wide range of industry sectors, including financial services, consumer products, industrial systems, and military applications. He holds numerous US and international patents covering UX and product design. He has been responsible for complex UX design solutions currently running on major world markets. Charles is the Chairman of the Design Protection Section of the Industrial Designers Society of America and is the IDSA liaison to the USPTO for Design Day 2011-12. He has lectured on UX design and usability at MIT, Stanford University and other leading educational institutions. Charles has been retained as an expert witness in over 50 major cases representing leading high technology companies covering UX and product design patent litigation. He has received awards and citations from the Human Factors and Ergonomics Society, Association of Computing Machines and, NASA. He has been quoted in Science, New York Times, BusinessWeek, Wall Street Journal. He writes on UX related trends on the blog Pulse>UX. He has a BS in Industrial Design from The Los Angeles Art Center College of Design and a Masters Degree in Ergonomics and Biomechanics from NYU. Mr. Mauro is also the founder and leader of the NYTECH event series “Designing the User Experience”.


http://www.mauronewmedia.com/

http://www.mauronewmedia.com/blog/2011/02/why-angry-birds-is-so-successful-a-cognitive-teardown-of-the-user-experience/

17

How To Think About UX Performance (Engagement Pyramid)

User cannot find critical features and cannot determine how to use them if they find them.

No Engagement

User can find critical features and understands how to use them but has no interest in doing so.

Low Engagement

User can find critical features and understands how to use them and has some interest in doing so.

Moderate Engagement

User can find critical features and understands how to use them and cannot stop doing so.

Deep Engagement

UX Testing

UX Testing

UX Testing

UX Testing

18

Methodology: Eye-Tracking

Business ProblemDo users see critical content and in what order?

DescriptionRespondents view content on a specialized workstation or glasses.

BenefitsVery accurate tracking of eye fixations and pathways.

LimitationsRelatively high cost, analysis is complex, data can be deceiving.

DataLive eye fixations, heat maps…etc.

Tools of ChoiceTobii - SMI



10

19

Methodology: Large-Sample Online Behavior Tracking

► Business problem► Major redesign of a large complex site that is business-critical?

► Description► 200-10,000+ respondents do tasks using online tracking / survey tools

► Benefits:► Large sample size, low cost per respondent, extensive data possible

► Limitations► No direct observation of users, survey design complex…other issues

► Data► You name it (data exports to professional analysis tools).

► Tools of Choice► Keynote WebEffective, UserZoom,



6


20

Recommended Reading: Albert and Tullis


21

Methodology: Lab-based UX Testing

► Business problem► Are there show-stopper (CI) usability problems with your user experience?

► Description► 12-24 Respondents undertake structured tasks in controlled setting (Lab)

► Benefits► Relatively fast, moderate cost, very graphic display of major issues

► Limitations► Small sample, study design, recruiting good respondents

► Data► Summary data in tabular and chart format PLUS video out-takes

► Tools ► Leased testing room, recruiting service and Morae (Industry Standard)



7


22

Recommended Reading: Tullis and Albert


23

Methodology: Professional Heuristics

► Business problem► Rapid feedback on UX design based on best practices or opinions

► Definition► “Heuristic is a simple procedure that helps find adequate, though often

imperfect, answers to difficult questions (same root as: eureka)”► Benefits

► Fast, low cost, can be very effective in some applications► Limitations

► No actual user data, analysis only as good as expert doing audit► Data

► Ranging from verbal direction to highly detailed recommendations► Tools of Choice

► Written or verbal descriptions and custom tools by each experts.Cost / respondent: NA


8


24

Methodology: Automated Online Card Sorting

► Business problem► User’s cannot understand where content they want is located?

► Description► Online card sorting based on terms you provide (or users create)

► Benefits► Large sample size, low cost, easy to field

► Limitations► Use of sorting tools confuse users, data hard to understand

► Data► Standard cluster analysis charts and more

► Tools of Choice► WebSort…and others



9


25

Methodology: fMRI (Brain Imaging)

► Business Problem?► What areas of the brain are being activated by UX design

► Description► Respondents given visual stimulus while in FMRI scanner

► Benefits► Maps design variables to core functions of the human brain

► Limitations► Expensive and data can be highly misleading

► Data► Brain scans

► Tools► Major medical centers and research services (some consultants)



10


26

Always Remember

When Testing Your User Experience

Relevance + Reliability = Success


nytech "measuring your user experience design"

Technology

research activities

rights reserved2

rights reserved5

director of research

research trends

longitudinal research

following data

description participants