THE FINAL DEBATE ON

How to Measure Usability & User Experience

2  

Our Speakers

Lee Cooper Head of UX Research, UserZoom Limited UK

Jeff Sauro Founding Principal,

measuringusability.com

Tim Bosenick Managing Director,

GfK Sirvaluse Consulting

Introduction by:

Javier Darriba Co Founder & Co CEO, UserZoom

#uzwebinar

3  

Quick Housekeeping

•  Chat box is available, if you have any questions

•  Or tweet your questions using #uzwebinar

•  There will be time for Q&A at the end of the webinar

•  We will be recording the webinar for future viewing

•  All attendees will receive a copy of the slides/recording

#uzwebinar

4  

About UserZoom

#1 Customer and User Experience Research and Analytics Platform

v  Remote Usability Testing

v  Voice of the Customer Studies

v  Information architecture research tools:

v Card Sorting v Tree Testing v Click testing

v  Survey Tool

Our products Our holistic approach

Listen

Measure Research

Test & act

5  

Highlighted Customers

#uzwebinar

6  

Agenda

•  UX measurement: Performance metrics Vs Satisfaction metrics – 5 mins

•  3 Ways of measuring the subjective part of UX

1. Scale usability system (SuS) – 10 mins 2. Supr – Q – 10 mins

3. UX Score – 10 mins

•  Q&A (10 min)

7  

User Experience Measurement

Performance Metrics

Satisfaction Metrics ?

-  Success rate -  Number of clicks -  Time on task -  …

-  SuS -  Supr-Q -  UX Score -  NPS

#uzwebinar

8  

Measuring Usability & UX: SuS Lee Cooper @userzoom_uk

9  

• What  is  it?  

• Where  did  it  come  from?  

• Why  do  I  like  it?  

SUS

#uzwebinar

10  

•  It’s  a  quesAonnaire  consisAng  of  ten  statements.  

•  Each  has  a  five-­‐point  scale  that  ranges  from  Strongly  Disagree  to  Strongly  Agree.  

•  There  are  five  posiAve  statements  and  five  negaAve  statements,  which  alternate.  

•  It  yields  a  single  usability  score  on  a  scale  of  0-­‐100.  

#uzwebinar

What is it?

11  

•  John  Brooke  1986,  Digital  Equipment  Co.  

•  “To  provide  us  with  a  measure  of  people’s  subjecAve  percepAons  of  the  usability  of  a  system”  

•  “To  allow  us  to  do  so  in  the  very  short  Ame  available  to  us  during  an  evaluaAon  session.”  

•  “It  was  probably  something  that  could  be  used  by  other  organizaAons…”  

John Brooke. SUS: A Retrospective (2013)

#uzwebinar

Where did it come from?

12  

•  It’s  Quick  &  Not-­‐So-­‐Dirty  •  It’s  Technology  AgnosAc  •  It’s  Free  •  It  Works  With  Small  Sample  Sizes  

•  It’s  Not  Perfect,  but…  •  It’s  Old  

#uzwebinar

Why do I like it?

13  

•  It’s  reliable  –  It  has  been  shown  to  be  more  reliable  than  many  homegrown  

and  commercially  available  quesAonnaires.    

•  It’s  valid  –  It  has  proven  to  be  effecAve  at  disAnguishing  between  usable  

and  unusable  systems  at  least  as  well  as,  and  o`en  beaer  than,  other  quesAonnaires.  

–  It  also  correlates  highly  with  other  quesAonnaire-­‐based  measurements  of  usability.  

Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011)

#uzwebinar

It’s Quick & Not-So-Dirty

14  

Microwave Bank: NCR Knowledge Lab

Symbian: UIQ/UIP Smartphones

3: 3G Services

Nokia: Smartphones

It’s Technology Agnostic

15  #uzwebinar

It’s Free

16  

•  It  works  with  small  and  large  sample  sizes.  

•  It  allows  you  to  measure  the  perceived  usability  of  a  system  with  a  small  sample  and  be  confident  that  you’ve  got  a  good  assessment.  

Tullis & Stetson. A Comparison of Questionnaires for Assessing Website Usability (2004)

#uzwebinar

It Works With Small Sample Sizes

17  

•  It’s  not  diagnosAc  –  It  does  not  tell  you  what  makes  a  system  usable  or  not.    

•  It  someAmes  confuses  –  Its  scores  are  not  percentages,  despite  returning  a  value  

between  0  and  100.    

•  Its  alternaAng  items  might  be  problemaAc  –  It  can  lead  to  responding  and  scoring  errors.  

Jeff Sauro. Measuring Usability With The System Usability Scale (SUS) (2011)

John Brooke. SUS: A Retrospective (2013)

#uzwebinar

It’s Not Perfect, but…

18  

•  There's  a  wealth  of  informaAon  about  its  use  along  with  a  body  of  normaAve  data.  

•  It  has  stood  the  test  of  Ame…  

Bangor, Kortum, and Miller. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale,”)(2009)

Jeff Sauro. Measuring Usability With The System Usability Scale (SUS)(2011)

#uzwebinar

It’s Old

19  

Measuring Usability & UX: Supr-Q Jeff Sauro @MeasuringU

Usability & Satisfaction SUPR-Q

Jeff Sauro | Measuring Usability LLC

20

21  Measuring Usability LLC #uzwebinar

ISO 9241 pt 11 Definition of Usability

22  

Usability + Credibility & Trust + Appearance + Loyalty

75 Candidate Items Measuring Usability LLC #uzwebinar

What makes an Effective Website?

23  

Usability This website is easy to use. It is easy to navigate within the website.

Credibility (Trust, Value & Comfort) This website keeps the promises it makes. I feel confident conducting business with this website.

Loyalty How likely are you to recommend this website to a friend or colleague? I will likely visit this website in the future.

Appearance I found the website to be attractive. The website has a clean and simple presentation.

NPS

Explains 94% of SUS

Standardized User Experience Percentile Rank-Questionnaire

Measuring Usability LLC #uzwebinar

SUPR-Q Items & Factors

24  

Retail Amazon eBay Zappos Apple Target JC Pennys Wal-Mart Hermes Boston Proper Container Store Crate & Barrel Crumpler Oriental Trading Sears LL-Bean Michaels Payless Shoes Pier 1 Imports Macys Netflix BestBuy OfficeMax OfficeDepot

Airlines United Airlines Frontier Airlines American Airlines Southwest Jet Blue Delta Frontier

3rd Party Travel Travelocity Expedia Orbitz Kayak

Government Illinois Colorado California CDC USA.gov New York State

Cell-Phone Carriers AT&T Wireless Verizon Sprint T-Mobile

Floral Service FTD 1800 Flowers ProFlowers

3rd Party Automotive Edmunds Kelly Blue Book Cars Autotrader

Financial Services PayPal Fidelity TD Ameritrade Vanguard Bellco US Bank Chase eTrade Scottrade Wells-Fargo

News & Information Wall Street Journal CNN FoxNews New York Times IMDB Yelp Craigslist Motley Fool Yahoo

Social Networking Facebook Twitter LinkedIn Flickr

Relationship/Dating Match.com eHarmony PlentyofFish.com

Hotels Hyatt Hilton

200+ Websites Total, 10,000 responses (30 to 200 per website) Measuring Usability LLC #uzwebinar

Selection of Websites in SUPR-Q Database

25  

A score of 50% = average score Measuring Usability LLC #uzwebinar

Normalized database

26  Measuring Usability LLC #uzwebinar

Scoring

27  

30% 41% 92% 38% 0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

Industry Average = 47%

SUPR-Q Health Care Industry

28  

Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance

Aetna 30% 22% 35% 33% 34%

BCBS 41% 26% 51% 45% 33%

Kaiser 92% 70% 91% 98% 79%

United Healthcare 38% 25% 48% 39% 33%

All Healthcare 47% 31% 55% 52% 40%

Overall, scores for the industry are a bit below average, with Kaiser being a clear leader, especially in trust and loyalty

Healthcare Websites

Measuring Usability LLC #uzwebinar

SUPR-Q subscale Scores

29  

85% 96% 66% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Industry Average =

82%

Measuring Usability LLC #uzwebinar

SUPR-Q Retail Banking

30  

Companies SUPR-Q Usability Loyalty Trust, Value & Comfort Appearance

Bank of America 66% 67% 70% 54% 52%

Chase Bank 96% 97% 93% 94% 83%

Wells Fargo 85% 88% 81% 84% 66%

All Banking 82% 84% 81% 77% 67%

Overall, scores for the industry are above average, with Chase Bank leading the pack, especially in usability and appearance

Retail Banking Websites

Measuring Usability LLC #uzwebinar

SUPR-Q subscale Scores

31  

 Feb  2011   Oct  2011    

NPS    73%    -­‐7%  

Credibility    99.8%    62%  

Usability    99.7%   73%  

Appearance    99.7%    80%  

Measuring Usability LLC #uzwebinar

Netflix Case Study

32  

Measuring  Usability  LLC  is  a  usability  and  quanAtaAve  research  firm  based  in  Denver,  Colorado  USA  focusing  on  the  staAsAcal  analysis  of  human  behavior  and  quanAfying  the  user  experience  for  Fortune  1000  companies.  

www.MeasuringUsability.com

Twitter @MeasuringU

August 20-22nd 2014 Denver, CO

Sponsored by UserZoom

DenverUX.com

Measuring Usability LLC #uzwebinar

Measuring Usability LLC

33  

Measuring Usability & UX: UX Score Tim Bosenick @GfK_UX

34  

Development 2011-2014

GfK UX Score

#uzwebinar © GfK 2014 | UX Score Development 34  

In 2011, GfK SirValUse and the GfK Verein conducted the basic research for the UX Score metric.

Desk research Operationalisation & Design Draft model of User Experience

Reduction to a core item set Eight different test objects 50 face-to-face interviews each

Final examination of the model Four different test objects 400 face-to-face interviews

Three qualities: task-oriented, self-oriented, aesthetic Standardised instrument for the measurement of UX

Validation study I Validation study II Preparation Model of UX

#uzwebinar © GfK 2014 | UX Score Development 35  

Task-oriented qualities Learnability / Operability

Self-oriented qualities Product Fit / Inspiration

Aesthetic qualities Look & Feel

Task-oriented qualities refer to the operation and learnability of the way in which the product works.

User-oriented qualities refer to feelings of belonging and the

stimulation of the user that the product engenders.

Product-related aspects refer to the features of the

product such as how it looks, feels and sounds.

These are the three dimensions of UX that we found and validated with the first and the second validation study in 2011.

#uzwebinar © GfK 2014 | UX Score Development 36  

In 2013, we conducted a third validation study in Germany, US and in China.

Collection of all relevant metrics to put in our questionnaire, like: SUS, NPS, AttraktDiff, Price Challenger Creation of item substitutes for the 'difficult' items.

The first part was the validation study in Germany. We conducted 100 face-to-face interviews of 60 minutes in our labs in Hamburg.

After the analysis of the German data confirmed the UX Score metric, User Centric and GfK China conducted the identical study in their countries.

After all the lab sessions had been conducted, we started with the online validation. The questionnaire was much shorter than the in the lab (see session structure).

Validation in Germany

Validation in the US and China

Preparation Online Validation (2014)

#uzwebinar © GfK 2014 | UX Score Development 37  

Test Stimuli with a standardized setup for all three countries

iPad 4 Samsung Galaxy Note 10.1

#uzwebinar © GfK 2014 | UX Score Development 38  

•  Introduction

•  Price Challenger pre-usage

•  Device 1 (tablets were shown in rotated order) •  Tryout part for around 10 minutes •  Brand items (including NPS) •  UX Score •  SUS •  AttraktDiff

•  Device 2 (repeat the tryout and the questions for the second device)

•  Price Challenger post-usage

Session Structure

#uzwebinar © GfK 2014 | UX Score Development 39  

UX Score: Overall results

The UX Score differentiates well between the devices in all three countries.

UX Score

Learnability

Operability

Product fit

Stimulation

Look and feel 5.4  

4.1  

4.4  

4.4  

5.0  

4.6  

USA (N=101) China (N=103) Germany (N=100)

4.6  

4.0  

4.3  

3.9  

4.4  

4.2  

5.3  

4.0  

4.3  

4.5  

5.2  

4.6  

5.2  

4.4  

4.7  

4.6  

5.0  

4.8  

5.2  

4.8  

4.8  

4.9  

5.1  

5.0  

4.5  

4.2  

4.1  

4.3  

4.8  

4.4  

iPad Samsung iPad Samsung iPad Samsung * *

* *

* *

* *

* *

* *

* *

* *

* *

* *

* *

* *

* Significant difference between the groups (significance level: 5%)

#uzwebinar © GfK 2014 | UX Score Development 40  

UX Score: Factor analysis

  DE USA CN   1 2 3 1 2 3 1 2 3

LE I  understand  the  terms  and  icons  used  by  my  tablet 0,819 0,145 0,183 0,864 0,023 0,202 0,845 0,251 0,175

LE I  can  learn  how  to  use  my  tablet  without  a  great  deal  of  thought 0,803 0,147 0,114 0,755 0,257 0,166 0,839 0,302 0,136

OP When  I  use  my  tablet,  it  always  does  what  I  want  it  to  do 0,773 0,312 0,217 0,651 0,523 0,065 0,426 0,761 0,232

OP Features  and  informaAon  are  always  where  I  expect  them  to  be 0,824 0,284 0,217 0,770 0,290 0,239 0,455 0,485 0,466

ST Using  this  tablet  is  inspiring  0,358 0,771 0,206 0,161 0,790 0,296 0,239 0,726 0,391

ST The  tablet  has  interesAng  new  features  0,065 0,847 0,169 0,075 0,844 0,072 0,225 0,829 0,189

PF The  tablet  is  right  for  me 0,459 0,568 0,363 0,410 0,703 0,298 0,420 0,672 0,428

PF The  tablet  has  exactly  the  features  I  need 0,450 0,586 0,281 0,379 0,733 0,166 0,354 0,720 0,377

LF I  like  the  look  of  the  tablet 0,215 0,304 0,830 0,120 0,439 0,751 0,150 0,332 0,887

LF The  tablet  looks  like  it  is  a  high-­‐quality  product 0,213 0,189 0,888 0,335 0,091 0,835 0,259 0,641 0,509

Total explained variance 74,5% 74,0% 78,6% Rotated component matrix. Extraction method: principle component analysis (Varimax with Kaiser normalization)

•  The item set performs very well in all three countries.

•  The explained variance of the UX Score metric is high.

•  The factor structure can also be reproduced in all three countries. However, the factor loadings for three items are in China not as clear as they are in the other two countries.

41  

Brand metrics: NPS

Promoters

Passive

Detractors

0  

1  

2  

3  

4  

5  

6  

7  

8  

9  

10  

NPS (Net Promoter Score) = promoters - detractors

47%  

19%  

29%  36%  

45%  45%  

36%  

48%  25%  

26%  

38%  

26%  

18%  

33%  

47%  

39%  

18%  

30%  

-15 -27 3 18 14 -31

iPad Samsung

Germany USA China

iPad Samsung

iPad Samsung

The NPS Score is different in all three countries. While the German and Chinese participants would rather recommend the iPad, the US participants are more into the Samsung Galaxy Note.

* Significant difference Lab (level: 5%) ** Significant difference Online (level: 5%)

© GfK 2014 | UX Score Development 42  

43 © GfK 2014 | UX Score Development

Explanation of the NPS by other global scales

None of the other metrics is able to explain as much of the NPS as the UX Score.

R² Dependent NPS DE USA CN

UX Score 0,517 0,584 0,565 SUS 0,322 0,422 0,308 AttrakDiff 0,228 0,476 0,356

Dependent: NPS

The diagram shows to which extent the Net Promotor Score (NPS) is explained by other global measures. An adjusted R² value of .50 and better can be interpreted as a relevant connection.

#uzwebinar 43  

44 © GfK 2014 | UX Score Development

Explanation of other global scales by the UX Score

Linear regression. Adjusted R² is diagrammed. The diagram shows to which extent the UX Score explains other global measures. An adjusted R² value of over .50 can be interpreted as a relevant connection.

R² Predictor UX Score DE USA CN

SUS 0,704 0,545 0,521 AttraktDiff 0,547 0,536 0,514 Intention to buy the brand 0,435 0,566 0,606 Intention to buy the product 0,457 0,572 0,636 Suitability of the brand 0,499 0,585 0,569 Positive feeling about the brand 0,469 0,460 0,568

Predictor: UX Score

The UX Score proved to be a valueable predictor for many global measures, like NPS, SUS etc.

In all three countries, the UX Score explaines the other global measures to a high extend.

#uzwebinar 44  

45 © GfK 2014 | UX Score Development

% Market Share // Results of the Price Challenger

This table shows the estimated market share from the Price Challenger tool. It shows the market share of both devices in relation to the height of the UX Score.

The results clearly indicate that a good user experience is important for a high market share.

The value of the UX Score can be used as a predictor for the market share. The higher the score, the higher the market share.

Germany USA China iPad Samsung iPad Samsung iPad Samsung

UX Score 0 – 3,9 points 4% 40% 2% 46% 41% 13% 4,0 – 4,4 points 10% 60% 17% 61% 57% 27% 4,5 – 4,9 points 25% 87% 21% 85% 59% 43% 5 – 6 points 51% 92% 39% 86% 72% 57%

#uzwebinar 45  

www.linkedin.com/company/userzoom

@UserZoom_UK

www.userzoom.co.uk

Contact us:

Cheshire (UK) 50a Alderley Road

Wilmslow, Cheshire SK9 1NT Phone: +44 (0) 1625 525 650

Any question?