incorporating pragmatic usability testing into a software test plan carla merrill, ph.d. focused...

Incorporating Pragmatic Usability Testing Into a Software Test Plan

Carla Merrill, Ph.D. Focused Designfocuseddesign.com

[email protected]

October 2005 Incorporating Pragmatic Usability Testing into a Software Test Plan © 2005 USA by Carla Merrill, Focused Design. 2

Presentation Topics

1. How important is usability for a software product?

2. Why is it hard to incorporate usability testing into test plans?

3. A solution through design

4. Integrating the solution into an agile process


The Importance of Usability

An interactive thought experiment for MS Word users

FIRST:Think of something you liked about the Word. Why do you like it? (because it does what you need)

What if I told you that the code implementing that feature was done in an old

programming language and was inelegant, even sloppy. Knowing that, think

about that feature again. Do you picture the code or the UI? (UI)Do you care about the structure of the code? (no)

SECOND: Think about something you don’t like about the product. Why don’t you like it? (because it doesn’t do what you need)

What if I told you that the code for that part of the product was extremely

elegantly written in a programming language du jour. Would that fact make

you like the feature? (no)


The Importance of Usability

Paradigm Shift:

1. Users define and evaluate a software product in terms of what they experience: the user interface.

2. Long-term success of a software product depends on how well the user interface meets the needs of target users.

3. Good-quality code doesn’t matter to users if the features it implements don’t meet their needs. Users don’t experience the code.


Perceived Difficulties of IncorporatingUsability Testing into a Software Test Plan

1. Specialized Human Factors skills required.

2. Specialized environments (test labs) required .

3. Appropriate test subjects required.

4. What is to be tested?

5. More project time and money required after standard test phase.

6. Test results and recommended changes are difficult to track and difficult to implement before product release.


Solution: Take Advantage of the Software Design Phase

Require the user interface design process to deliver user scenarios that can be added to test plans.


Required Format for User Scenarios

1. Pragmatic: only scenarios that capture the primary tasks of your target users (contrast with use cases).

2. Created from the user’s perspective with user-based vocabulary that is implementation free.

3. Step-by-step format, limited to 10 steps per scenario.

4. Long tasks are chunked into a series of scenarios that are prioritized by importance to target users.

5. Scenarios are validated with users before testing begins.


Adding Scenarios To Test Plans

1. Tester can define a test procedure for each scenario by adding implementation language.

2. Tester can evaluate success/failure of a scenario by standard test methods: (time, system failure points, UI failure points, number of implementation steps required to complete the scenario).

3. Tester can log bugs for failure points in the test procedure.


Advantages of Incorporating User Scenarios: Usability Can Be Tested!

1. Testers have the skills.

2. Usability tests are clearly defined.

3. Standard test procedures are used.

4. Testing cycle allows usability bugs to be fixed before release.

5. Usability bugs can be triaged appropriately based on priority ratings of scenarios.

6. Usability of the product can be measured using standard testing metrics.


The Goal-Oriented Design Solution (GOD-S)

Note: Slides 11-15 will not be discussed in this presentation. They are included for reference only.


User Profiling: What do we need to know about users?


User Profiling: How do we gather the information we need?

Gain access to users

Meet users in the right environment

Prepare for the meetings

Practice Guided Listening™

Create user reports


Persona and Scenario Development: How do we make effective use of the information we’ve gathered?

The major challenges of UI design “Many users — one user interface” Feature-creep and elastic users Who’s to say what users want?

What is a persona?*

What are scenarios?

A persona with scenarios can help solve the problems of UI design

* Based on the usage of the term coined by Alan Cooper


Step 1. Construct a Matrix from User Profiles

Step 2. Analyze the Matrix

Step 3. Build Goal-Oriented Scenarios for a Persona

Persona and Scenario Development: How do we create a persona and scenarios?


User Interface Design: Design the User Interface Based on Goal-Oriented Scenarios

A sound strategy for UI design: Top-down design – follow the user

1 step at a time – focus on enabling scenarios

There is no “right answer” – verify the designwith users

It’s creative

It requires the UI design skillset

It’s iterative and needs scheduled time in the project


How does GOD work with different development models?

How to merge GOD with a Development Model

Serial Development Model

Agile Development Model


Serial Development ModelC

om

ple

ten

ess,

Sta

bilit

y

Time

Requirements

Testing

Design

Development


Agile Development Model ( after Alistair Cockburn, http://alistair.cockburn.us )

Com

ple

ten

ess,

Sta

bilit

y

Time

Requirements

Testing

Design

Development


Potential Problems of the Merger

Agile Development

• Bottom–up coding – don’t need the UI framework

• Use cases focus on system needs

• Code chunks are pieces isolated from the whole application

• UI chunks designed in context of a UI framework

• UI chunks based on user taskflows

• UI design is always ahead of the code

GOD• Top-down design – a UI framework first

• Scenarios focus on primary needs of a person

• Code chunks are based on programming methodology

• Coding of UI is simpler, less critical, and done last – the “lipstick”


Potential Synergies of the Merger

Agile Development• Bottom–up coding allows rapid progress, while absorbing thin topcoat

• Use cases can focus on user needs, but care for system needs

• Code chunks allow for iteration, absorbing usability feedback

• Top-down design cares for the Big Picture

• Scenarios keep the focus on user needs

• The Big Picture design can provide a logical scheme for chunking

GOD


Strategies and Tactics for a Successful Merger

• Build UI design prework into the process

• Create persona/scenarios

• Create UI framework/primary navigation (inch-deep, top-down coding)

• Get buy-in on UI framework from stakeholders

• Create chunks based on UI framework

• Continue inch-deep top-down design and coding in each iteration

• Create user-focused use cases for chunks

• Include usability testing in each iteration and absorb the feedback

• Enable teaming between UI designers and UI coders

• Require UI design realism: designers use most efficient GUI controls

• Bleed UI deficits into the next iteration until UI is complete

Agile Development + GOD


Timeline for a Successful Agile + GOD Project

Project start Product delivery

High-level business requirements

GOD prework

Schedule/chunking

Team definition

System framework design

Chunking refinement as needed

Agile iterations 1…n

with multiple teams Integration

Usability testingBeta


Timeline for a Successful Agile + GOD Iteration

V starts End of V’s final QA

GOD usability testing

Reflections mtg

Business reqs

Chunk definition

GOD V prework GOD wireframing

DEV post V cleanup/integration DEV coding

GOD UI deficit list

QA post V bug tracking

Use case development

QA new V test scripting

= code drop

QA testing

Start the next V

= code test


Summary

1. Usability testing is important for the success of a software product.

2. Pragmatic usability testing can be incorporated into the standard test plan through user scenarios developed in the design phase.

3. Testing can be done using existing QA resources and bug tracking systems.

4. Goal-Oriented Design supports pragmatic usability testing in an agile software development environment.

incorporating pragmatic usability testing into a software test plan carla merrill, ph.d. focused...

Documents

focused design

software product

software design phaserequire

test results

standard test phase

appropriate test subjects

user interface design

users dont