Usability Modeling and Measurement

Philip Lew

www.xbosoft.com

Agenda

• Introduction– The importance of usability – Specifics for the web / mobile

• What is Usability?• Usability Modeling and Measurements• Case study• Conclusion• Next Steps

Importance of Usability

• Usability is important especially for web applications (SaaS, and websites) – And now especially mobile applications

• Most prevalent development delivery model today

• Without good usability:– Users will leave the applications– For mobile, if they can’t learn in 30 seconds,

they won’t come back

Why is Usability Important?

Web and Mobile have Changed the Terrain

• Business models have changed– Instead of paying

upfront and ‘owning’ the software

– Pay as you go, pay by subscription

• Behavior and expectations have changed

Many Alternatives• Depending on the

type of application– Users could leave

and go elsewhere– Simply not use your

mobile app (if only providing alternative access to a main application)

Thin Client DeploymentMost Prevalent Development Delivery Model Today

• Mobile and Cloud are singing the same song• Service delivery versus product delivery

What is Usability

In relation to qualityIn relation to user experience

General Thoughts on Usability

• Understandability• Learnability• Operability• Attractiveness• Navigation• Responsiveness-performance• Efficiency

Usability in Quality Standards• How is usability

defined?– Standard

definitions– Many others

who say similar things

• Usability– Component

of quality as listed in many standards

Standard DefinitionISO 9126-1(2000)

The capability of the software product to be understood, learned, used, and attractive to the user, when used under specified conditions.

ISO 9241-11(1998)

The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.

IEEE 600.12(1990)

The ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component.

Seffah [33]

Usability Model Comparisons

Current Research / Usability Standards

Kappel et al., The Discipline of Systematic Development of Web Applications, 2003, John Wiley

and Sons.

Current Research – Usability Standards

Quality

Usability

Satisfaction

ISO9126-1

ISO 9241-11

IEEE 610

IEEE XXXYISO 25010

Bigger Picture Quality From ISO point of view

ISO 25010CMMI

Usa

bilit

y

Usa

bilit

y

Usability as a Key Characteristic of Product Quality

Source: ISO 25010

Source: ISO 25010

What is Usability-Effect of the Software Product

Degree to which specified users can achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

Source: ISO 25010

What is Usability-EffectIn Actual Usage

• Effectiveness– The degree to which specified users can achieve specified goals with

accuracy and completeness in a specified context of use.• Efficiency

– The degree to which specified users expend appropriate amounts of resources in relation to the effectiveness achieved in a specified context of use.

– NOTE Relevant resources can include time to complete the task, materials, or the financial cost of usage.

• Satisfaction– The degree to which users are satisfied in a specified context of use.

Satisfaction is further subdivided into sub-characteristics:• Likability (cognitive satisfaction)• Pleasure (emotional satisfaction)• Comfort (physical satisfaction)• Trust

Usability in Actual Usage

• User role• Objective• Task• Environment• Domain• …

specified users

specified goals

specified context of use

What else can you think of?

Defining Usability For Your Organization

• Attributes expressed hierarchically• Any number of sub-levels is OK• Achieving the sub-attributes=achieving the high level

attribute -measurable

Quality

Attribute 1

Characteristic 1 Characteristic 2 Characteristic n

Subcharacteristic 1 Subcharacteristic 2 Subcharacteristic n

Attribute 2 Attribute 3 Attribute 1

Usability

Let’s Define UsabilityFrom the Product Viewpoint

Usability

Save Location

Characteristic 1 Characteristic 2 Characteristic n

Subcharacteristic 1 Subcharacteristic 2 Subcharacteristic n

Attribute 1 Attribute 2 Attribute n

UnderstandableNavigation

Control Stability Previous-Next

Home Location Ease of finding

Bigger Picture on Usability

• Can be measured from the design point of view or of the product

• Can be measured ‘in-use’ with real users

Usa

bilit

y

Usa

bilit

y

Defining Usability from an Effect-Real usage Point of View

Quality

Errors

Effectiveness Efficiency Characteristic n

Accuracy Completeness Subcharacteristic n

Attribute 2 Attribute 3 Attribute 1

Usability

Satisfaction

Other Potential Attributes Measurements for ‘in use’ Usability

• Effectiveness– Completion rates– Error rate– Help usage

• Efficiency– Task time– Backtracking

• Learnability– Learning rate– Task time deviation

Developing a CombinedUsability Model

Application Design and Environment

User Behavior and Activity

Measurable and quantifiable factors

Measured ‘in-use’ Usability

External Quality Requirements (for Shopping Cart Entity) 1 Usability 1.1 Understandability 1.1.1 Icon/label ease to be recognized 1.1.2 Information grouping cohesiveness 1.2 Learnability 1.2.1 ……………………………………………………….. 1.3 Operability 1.3.1 Control permanence 1.3.2 Expected behaviour of Controls 2 Content Quality 2.1 Content Suitability 2.1.1 Basic Information Coverage 2.1.1.1 Line item information completeness 2.1.1.2 Product description appropriateness 2.1.2 Coverage of other Contextual Information 2.1.2.1 …………………………………………………………

Example of Product Quality Model with Attributes

(Operability in ISO 25010)

Measurable Attributes

• Attribute name• Description and purpose• How to measure• What is measured• Measurement/Calculation• Range (min, max)• Objective• Current

Once you have a model (what you are going to measure), then you start doing IT!

Usability Measurement

27

Attribute Scale How Measure orCalculation

Objective Current

Help completeness

Percent ofMenu items with help

Comparemenus andhelp items

% 90% 40%

Ease ofaccess

Keystrokes tofind/use a feature/function/information

Sample 50items

% 3 12

Consistency Number locations for same button

Examinemenus anddoc.

integer 1 5

Accuracy Number reportederrors

Collect from log files

Integer or % <5 10

Usability Measurement Methods

Let’s get started

Logging

Heuristic Evaluation

Labs

Focus groups

WalkThroughs

Satisfaction Surveys

Current Research-Summary of Usability Evaluation Methods

Usability LoggingMeasurement and Data Collection

• Identify users by using session ID to identify a unique user.

• Iteratively insert code into the application

• Collect data• Analyze the data for each

attribute in different dimensions and aggregations

• Determine the need for further calculations and what attributes to measure further

• Revise the data we are collecting, adding or decreasing granularity

Satisfaction Surveys

The process:1. Calculate the usability score (satisfaction) of version X2. Do the survey3. Change to version X.1 – make changes to the

software directly correlated to the usability factors to either increase or decrease the usability score

4. Do the survey again 5. See if differences made change impact the survey

results

Notes on Satisfaction and Usability

• Satisfaction is a subjective feeling dependent on many things other than usability:– A user can be highly

satisfied but the application with low usability.

– An application can be highly usable (high usability) but the user is not satisfied!

Highly usablesoftware

My passworddoesn’t work

don’t havewhat I want I’m unsatisfied

I’m satisfied!

!low usability

software

Finished my work today

Nice weather today

External Quality Requirements Measure EI value P/GI value Global Quality Indicator 61.97%

1 Usability 60.88% 1.1 Understandability 83% 1.1.1 Icon/label ease to be recognized 100% 1.1.2 Information grouping cohesiveness 66% 1.2 Learnability 51.97% 1.2.1 ……………………………………………… … 1.3 Operability 49.50% 1.3.1 Control permanence 100% 1.3.2 Expected behaviour 50% 2 Content Quality 63.05% 2.1 Content Suitability 63.05% 2.1.1 Basic Information Coverage 50% 2.1.1.1 Line item information completeness 2 50% 2.1.1.2 Product description appropriateness 50% 2.1.2 Coverage of other Contextual Information 76.89% 2.1.2.1 ……………………………………………….. … 2.1.2.2 Return policy information completeness 33%

Sample Evaluation

Healthcare SoftwareHeuristic Evaluation

Conclusion

• Usability is an abstract concept• Defining usability is different for each

organization• Need a model for your organization• The model is the foundation of what to

measure• Once you can measure, then you can evaluate

and improve

Next Steps

• Produce an action plan– What usability attributes are important to your

organization?• Develop a model

– What data can you collect/Which technique can you use

• Maybe some elements of the model drop out-can’t be measured that easily

– Start collecting and developing benchmarks• Discuss with your manager/team

Thanks

Questions and Answers