discount evaluation evaluating with experts. discount evaluation techniques basis: – observing...
TRANSCRIPT
Discount Evaluation
Evaluating with experts
Discount Evaluation Techniques
Basis:– Observing users can be time-consuming and expensive– Try to predict usability rather than observing it directly– Conserve resources (quick & low cost)
Approach: – HCI experts interact with system and try to find potential
problems and give prescriptive feedback– Best if
Haven’t used earlier prototype Familiar with domain or task Understand user perspectives
Heuristic Evaluation
Several expert usability evaluators assess system based on simple and general heuristics (principles or rules of thumb)
Mainly qualitative use with experts Predictive
Developed by Jakob Nielsen(Web site: www.useit.com)
Procedure
1. Gather inputs
2. Evaluate system
3. Debriefing and collection
4. Severity rating
1: Gather Inputs
Who are evaluators?– Need to learn about domain, its practices
Get the prototype to be studied– May vary from mock-ups and storyboards to a
working system
How many experts?
Nielsen found thatabout 5 evaluations found 75% of the problems
Above that you get more, but at decreasing efficiency
2: Evaluate System
Reviewers evaluate system based on high-level heuristics.
Where to get heuristics?– http://www.useit.com/papers/heuristic/– http://www.asktog.com/basics/firstPrinciples.html
Heuristics
use simple and natural dialog
speak user’s language minimize memory load be consistent provide feedback
provide clearly marked exits
provide shortcuts provide good error
messages prevent errors
Neilsen’s Heuristics
visibility of system status aesthetic and minimalist
design user control and freedom consistency and standards error prevention
recognition rather than recall flexibility and efficiency of
use recognition, diagnosis and
recovery from errors help and documentation match between system and
real world
Process
Perform two or more passes through system inspecting– Flow from screen to screen– Each screen
Evaluate against heuristics Find “problems”
– Subjective (if you think it is, it is)– Don’t dwell on whether it is or isn’t
3: Debriefing
Organize all problems found by different reviewers– At this point, decide what are and aren’t problems– Group, structure– Document and record them
4: Severity Rating
Based on– frequency– impact– persistence– market impact
Rating scale– 0: not a problem– 1: cosmetic issue, only fixed if extra time– 2: minor usability problem, low priority– 3: major usability problem, high priority– 4: usability catastrophe, must be fixed
Advantages
Few ethical issues to consider Inexpensive, quick
Getting someone practiced in method and knowledgeable of domain is valuable
Challenges
Very subjective assessment of problems– Depends of expertise of reviewers
Why are these the right heuristics?– Others have been suggested
How to determine what is a true usability problem– Some recent papers suggest that many identified
“problems” really aren’t
Your turn:
Banner – adding a course
Use Nielsen’s heuristics (p 686) List all problems Come up to the board and put up at least one
new one We’ll rate as a group
Neilsen’s Heuristics
visibility of system status aesthetic and minimalist
design user control and freedom consistency and standards error prevention
recognition rather than recall flexibility and efficiency of
use recognition, diagnosis and
recovery from errors help and documentation match between system and
real world
Heuristic Evaluation: review
Design team provides prototype and chooses a set of heuristics
Experts systematically step through entire prototype and write down all problems
Design team creates master list, assigns severity rating
Design team decides how to modify design
Cognitive Walkthrough
More evaluating with experts
Cognitive Walkthrough
Assess learnability and usability through simulation of way users explore and become familiar with interactive system– Qualitative– Predictive– With experts
to examine learnability and novice behavior
From Polson, Lewis, et al at UC Boulder
CW: Process
Construct carefully designed tasks from system spec or screen mock-up
Walk through (cognitive & operational) activities required to go from one screen to another
Review actions needed for task, attempt to predict how users would behave and what problems they’ll encounter
CW: Assumptions
User has rough plan User explores system, looking for actions to
contribute to performance of action User selects action seems best for desired
goal User interprets response and assesses
whether progress has been made toward completing task
CW: Requirements
Description of users and their backgrounds Description of task user is to perform Complete list of the actions required to
complete task Prototype or description of system
CW: Methodology
Experts step through action sequence– Action 1– Response A, B, ..– Action 2– Response A– ...
For each one, ask four questions and try to construct a believability story
CW: Questions
1. Will users be trying to produce whatever effect action has?
2. Will users be able to notice that the correct action is available? (is it visible)
3. Once found, will they know it’s the right one for desired effect? (is it correct)
4. Will users understand feedback after action?
Let’s practice: My Internet Radio
User characteristics– Technology savy users– Familiar with computers– Understand Internet radio concept– Just joined and downloaded this radio
Task: add a station to presets
Click genre Scroll list and choose genre Assuming station is on first page, add station
to presets -- right-click on station, choose add to presets from popup menu.
Click OK on Presets
Task:Click – Pick a genre
1. Will users be trying to produce whatever effect action has?
2. Will users be able to notice that the correct action is available?
3. Once found, will they know it’s the right one for desired effect?
4. Will users understand feedback after action?
CW: Answering the Questions
1. Will user be trying to produce effect?– Typical supporting evidence
It is part of their original task They have experience using the system The system tells them to do it
– No evidence? Construct a failure scenario Explain, back up opinion
CW: Next Question
2.Will user notice action is available?– Typical supporting evidence
Experience Visible device, such as a button Perceivable representation of an action such as a menu
item
CW: Next Question
3.Will user know it’s the right one for the effect?– Typical supporting evidence
Experience Interface provides a visual item (such as prompt) to
connect action to result effect All other actions look wrong
CW: Next Question
4.Will user understand the feedback?– Typical supporting evidence
Experience Recognize a connection between a system response
and what user was trying to do
Scroll list and choose genre
1. Will users be trying to produce whatever effect action has?
2. Will users be able to notice that the correct action is available?
3. Once found, will they know it’s the right one for desired effect?
4. Will users understand feedback after action?
Action: Right click on station and choose “Add to Presets”
1. Will users be trying to produce whatever effect action has?
2. Will users be able to notice that the correct action is available?
3. Once found, will they know it’s the right one for desired effect?
4. Will users understand feedback after action?
Action: Click OK
1. Will users be trying to produce whatever effect action has?
2. Will users be able to notice that the correct action is available?
3. Once found, will they know it’s the right one for desired effect?
4. Will users understand feedback after action?
Problems
Did I pick the right task? Or list out the right sequence of actions?
CW Summary
Advantages Explores important
characteristic of learnability
Novice perspective Detailed, careful
examination Working prototype not
necessary
Disadvantages Can be time consuming May find problems that
aren’t really problems Narrow focus, may not
evaluate entire interface
Your turn
Banner – adding a class
What are our tasks?
What are the actions?
CW: Questions
1. Will users be trying to produce whatever effect action has?
2. Will users be able to notice that the correct action is available? (is it visible)
3. Once found, will they know it’s the right one for desired effect? (is it correct)
4. Will users understand feedback after action?
CW: responsibilities
Design team creates prototype, user characteristics Design team chooses tasks, lists out every action
and response Experts answer 4 questions for every
action/response Design team gathers responses and feedback Design team determines how to modify the design
Next Wednesday’s plan
3:30-4:05 Cognitive Walkthrough 1 member of design team (D) will facilitate 4
members of the evaluating team D walks through each task, evaluating team
asks and answers the 4 questions for every action
D makes sure all feedback is written down and takes back to rest of design team
Cognitive Walkthrough pairs
Rabid Bunnies – User One The Team - Satisfaction Inventive Innovators – No Spoon Awesome - NoName
Monday’s plan
4:10-4:45 Heuristic evaluation 1 member of the Design team (D2) demos
the prototype to the evaluating team D2 provides the set of heuristics Evaluating team individually writes down all
problems they see and gives it to D2
Heuristic evaluation teams
Rabid Bunnies – The Team User Won - Satisfaction Inventive Innovators - Awesome NoSpoon – NoName
Come Prepared!
Bring your prototype, have it ready to go at 3:30
Choose facilitator (D1 & D2) for both Bring task & action lists for cognitive
walkthrough Bring heuristics for heuristic evaluation As an evaluator – be detailed, thorough, and
constructive