usability testing: analyzing data

• Compile

• Summarize data

• Analyze data

• Develop recommendations

• Produce the final report

Wednesday, March 28, 12

Data Analysis

Wednesday, March 28, 12

next 2 weeks:

Wednesday, March 28, 12

Wednesday, March 28, 12

Wednesday, March 28, 12

Step 1: Compile

Wednesday, March 28, 12

Wednesday, March 28, 12

Get it into a pile.

Wednesday, March 28, 12

Get your piles sorted.

Wednesday, March 28, 12

Ideally:

Wednesday, March 28, 12

compile as you test.

Wednesday, March 28, 12

further test the test.

Wednesday, March 28, 12

keep the data legible.

Wednesday, March 28, 12

• Type handwritten notes

• Verify data captures

• Back up files

Wednesday, March 28, 12

Organize raw data.

Wednesday, March 28, 12

Do whatever it takes to get it done.

Wednesday, March 28, 12

• lists

• tallies

• matrices

• stories

• structure models

• flow diagrams

Wednesday, March 28, 12

Wednesday, March 28, 12

Wednesday, March 28, 12

Wednesday, March 28, 12

Wednesday, March 28, 12

Goal:

Wednesday, March 28, 12

Wednesday, March 28, 12

Step 2: summarize

Wednesday, March 28, 12

1. task accuracy

Wednesday, March 28, 12

number of errors made per task

Wednesday, March 28, 12

• categorize errors by type

• ommission

• commission

Wednesday, March 28, 12

# of participants who performed:

• successfully

• successfully but with some assistance

• assistance types, tasks, products

Wednesday, March 28, 12

Vital Statistics:

Wednesday, March 28, 12

% performing successfully, including those who needed assistance

• regardless of time limits

• regardless of assistance needed

Wednesday, March 28, 12

If this percentage is very low, you’ve got a

serious problem.

Wednesday, March 28, 12

% performing successfully

• including those who corrected errors

Wednesday, March 28, 12

% performing successfully within benchmark

• indicative of efficiency

Wednesday, March 28, 12

2. task timings

Wednesday, March 28, 12

Wednesday, March 28, 12

summarize preference data

Wednesday, March 28, 12

Limited choice questions:

• Sum by choice

• Chart trends

Wednesday, March 28, 12

Free-form questions:• Group into meaningful categories

• Ex: particular screen, doc section, etc.

Wednesday, March 28, 12

Summarize other measures

• Unnecessary returns to main navigation

• Number/type of hints and prompts

• Times site map was accessed

• Points of hesitation (and for how long)

Wednesday, March 28, 12

Summarize by Group/Version

Wednesday, March 28, 12

(see pages 255 - 259)

Wednesday, March 28, 12

3: analyze data

Wednesday, March 28, 12

What didn’t succeed?

Wednesday, March 28, 12

Wednesday, March 28, 12

What were the difficulties?

Wednesday, March 28, 12

What were the error sources?

Wednesday, March 28, 12

Prioritize problemsby severity

Wednesday, March 28, 12

Severity descriptions & definitions: p. 262

Wednesday, March 28, 12