an introduction to usability testing bill killam, ma chfp adjunct professor university of maryland...
TRANSCRIPT
![Page 1: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/1.jpg)
An Introduction to Usability Testing
Bill Killam, MA CHFPAdjunct Professor
University of [email protected]
User-Centered Design www.user-centereddesign.com
![Page 2: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/2.jpg)
Background
User-Centered Design www.user-centereddesign.com
![Page 3: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/3.jpg)
3
Definitions
“Usability testing” is the common name for multiple forms both user and non-user based system evaluation focused on a specific aspect of the design
Done for many, many years prior, but popularized in the media by Jakob Neilson in the 1990’s
User-Centered Design www.user-centereddesign.com
![Page 4: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/4.jpg)
4 User-Centered Design www.user-centereddesign.com
Usability Defined
Accessibility– A precursor to usability: if users cannot gain access to the product, its
usability is a moot point Functional Suitability
– Does the product contain the functionality required by the user? Functional Discoverability
– Can the user “discover” the functions of a product? Ease-of-learning
– Can the user figure out how to exercise the functionality provided? Ease-of-use
– Can the user exercise the functionality accurately and efficiently once its learned (includes accessibility issues)?
– Can users use it safely? Ease-of-recall
– Can the knowledge of operation be easily maintained over time? Safety
– Can the user operate the system in relative safety, and recover from errors?
Subjective Preference– Do user’s like using it?
![Page 5: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/5.jpg)
5 User-Centered Design www.user-centereddesign.com
What does “usability” mean?
ISO 9126 – “A set of attributes that bear on the effort needed
for use, and on the individual assessment of such use, by a stated or implied set of users”
ISO 9241– “Extent to which a product can be used by
specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.”
![Page 6: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/6.jpg)
6 User-Centered Design www.user-centereddesign.com
What does “usability” mean? (concluded)
Jakob Neilson– Satisfaction– Efficiency– Learnability– Low Errors– Memorability
Ben Shneiderman– Ease of learning – Speed of task completion – Low error rate – Retention of knowledge over time – User satisfaction
![Page 7: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/7.jpg)
Usability Testing is…
Any of a number of methodologies used to try to determine how a product’s design contributes or hinders its use when used by the intended users to perform the intend tasks in the intended environment
Most common forms include– Modeling & analysis– Expert Review/Heuristic Evaluations– User-based testing
User-Centered Design www.user-centereddesign.com
![Page 8: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/8.jpg)
When is usability assessed?
On an existing product to determine if usability problems exist
During the design phase of a product During the development phase of a product to
assess proposed changes Once a design is completed to determine if goals
were met
User-Centered Design www.user-centereddesign.com
![Page 9: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/9.jpg)
Methods
User-Centered Design www.user-centereddesign.com
![Page 10: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/10.jpg)
10
Types of Usability Testing
Modeling & Analysis– Link Analysis– Time Line Analysis– GOMS Modeling
Non-User Based Testing– Compliance Reviews– Expert Reviews (a.k.a. Heuristic Evaluations)– Cognitive Walkthrough– Pluralistic Walkthroughs
User-Centered Design www.user-centereddesign.com
![Page 11: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/11.jpg)
11
Types of Usability Testing (concluded)User-based Testing
– User Surveys/Questionnaires– Interviews– “Contextual Inquiry” (Field Observation)– Think Aloud Protocol– Interrupted-task Based Protocol– Co-Discover Protocol– Performance-based Protocol
User-Centered Design www.user-centereddesign.com
![Page 12: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/12.jpg)
Non-User Based Testing
User-Centered Design www.user-centereddesign.com12
![Page 13: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/13.jpg)
13
Compliance Testing
Possible (within limits) to be performed by anyone
Can remove the low level usability issues that often mask more significant usability issues
User-Centered Design www.user-centereddesign.com
![Page 14: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/14.jpg)
14
Compliance Testing (concluded)
Style Guide-based Testing– Checklists– Interpretation Issues– Scope Limitations
Available Standards– Commercially GUI & Web Standards and Style
Guides– Domain Specific GUI & Web Standards and
Style Guides– Internal Standards and Style Guides
Interface Specification Testing– May revert to user acceptance testing
User-Centered Design www.user-centereddesign.com
![Page 15: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/15.jpg)
15
Pluralistic Walkthrough
Team ApproachBest if a diverse population of reviewers Issues related to cognition (understanding)
more than presentationAlso low cost usability testing Highly dependent on the qualifications of
the reviewer(s)
User-Centered Design www.user-centereddesign.com
![Page 16: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/16.jpg)
16
Cognitive Walkthrough
Specific review to ensure the correct information is available for the task being performed
Also low cost usability testing Highly dependent on the qualifications of
the reviewer(s)
User-Centered Design www.user-centereddesign.com
![Page 17: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/17.jpg)
17
Expert Review
Aka: Heuristic EvaluationOne or more usability experts review a
product, application, etc.Free format review or structured reviewSubjective but based on sound usability
and design principlesHighly dependent on the qualifications of
the reviewer(s)
User-Centered Design www.user-centereddesign.com
![Page 18: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/18.jpg)
18
Expert Review (Concluded)
Nielson’s 10 Most Common Mistakes Made by Web Developers (three versions)
Shneiderman’s 8 Golden RulesConstantine & Lockwood HeuristicsForrester Group HeuristicsNorman’s 4 Principles of Usability
User-Centered Design www.user-centereddesign.com
![Page 19: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/19.jpg)
1st HeuristicFunctional discoverability through obvious
interactive elements and adequate feedback
User-Centered Design www.user-centereddesign.com19
![Page 20: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/20.jpg)
User-Centered Design www.user-centereddesign.com20
![Page 21: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/21.jpg)
User-Centered Design www.user-centereddesign.com21
![Page 22: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/22.jpg)
2nd Heuristic
A good, complete, and unambiguous cognitive (or conceptual) model to predict the effects of our
actions
User-Centered Design www.user-centereddesign.com22
![Page 23: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/23.jpg)
23 User-Centered Design www.user-centereddesign.com
![Page 24: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/24.jpg)
24 User-Centered Design www.user-centereddesign.com
![Page 25: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/25.jpg)
25 User-Centered Design www.user-centereddesign.com
![Page 26: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/26.jpg)
26 User-Centered Design www.user-centereddesign.com
![Page 27: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/27.jpg)
27
Cognitive Models
We all develop cognitive models– They may not be complete– They may be inconsistent– They ay be self contradicting– They are not always correct
We don’t necessarily invest in maintaining our cognitive models
User-Centered Design www.user-centereddesign.com
![Page 28: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/28.jpg)
28 User-Centered Design www.user-centereddesign.com
Cognitive Model Issues
![Page 29: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/29.jpg)
Conceptual Model Issues
29
![Page 30: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/30.jpg)
3rd HeuristicDesign for the intended users (and not
yourself)
User-Centered Design www.user-centereddesign.com
![Page 31: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/31.jpg)
31 User-Centered Design www.user-centereddesign.com
![Page 32: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/32.jpg)
32 User-Centered Design www.user-centereddesign.com
![Page 33: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/33.jpg)
33 User-Centered Design www.user-centereddesign.com
1131 SAN 0820+1 LGW AA 2734 FCYBM D10 1
AA 2734 CHG PLANE AT DFW
X12 1805 SAN 1425+1 LGW BA 284 FJMSB D10 1
2100 SAN 2030+1 LHR TW 702 FCYBQ * 2
TW 702 EQUIPMENT 767 LAX L-10
![Page 34: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/34.jpg)
34 User-Centered Design www.user-centereddesign.com
Midnight Midnight6:00 AM 6:00 AMNoon Noon6:00 PM 6:00 PM
TWA 702
BA 284
AA 2734Gatwick
Gatwick
Heathrow
(7:00 AM)(1:00 PM) (7:00 PM) (1:00 AM)
Local(London)
![Page 35: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/35.jpg)
4th HeuristicDesign for Errors (Slips)
User-Centered Design www.user-centereddesign.com35
![Page 36: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/36.jpg)
36
Error versus Slip
Errors are generated by a lack of understanding or a lack of sufficient or correct information– Lack of sufficient or correct information is the
responsibility of the designer in the presentation layer of an interface
– Lack of understanding is the responsibility of the designer in interaction and in conceptual model of an interface
– Errors are often undetectable by the end user– Examples: Refrigerator, Ollie North
Slips are common users issues– Hand/eye coordination or basic control of our psychomotor
systems– Exacerbated by distraction, speed, attention overload– Unavoidable by design but need to be anticipated and
addressed by the designerUser-Centered Design www.user-centereddesign.com
![Page 37: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/37.jpg)
User Based Testing
User-Centered Design www.user-centereddesign.com37
![Page 38: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/38.jpg)
Statistics: A Primer
User-Centered Design www.user-centereddesign.com38
![Page 39: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/39.jpg)
Why Statistics?
Testing is used to support a decision– For example, “this design change is going to be
better for users”, or “design A is better than design B”
Research to used to test a hypothesis based on a theory– Smoking increases the likelihood of developing
cancer
Usability testing is generally done with small samples (mostly do to the cost associated with any alternatives)
Statistics are used to provide a way relate the small sample tested to the larger population
All statistical analysis assumes the data obtained is valid and reliable
User-Centered Design www.user-centereddesign.com39
![Page 40: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/40.jpg)
Validity
Validity is the degree to which the results of a research study provide trustworthy information about the truth or falsity of the hypothesis*
Internal validity refers to the situation where the “experimental treatments make a difference in this specific experimental instance” (from Cambell, D.T. & Stanley, J.C. (1963) Experimental and Quasi-experimental Designs for Research
External validity asks the question of “generalizability”
User-Centered Design www.user-centereddesign.com40
*Cherulnik, P.D. 2001. Methods for Behavioural Research: A Systematic Approach
![Page 41: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/41.jpg)
Reliability
Reliability is the ability of a test to show the same results if conducted multiple times– Test-retest reliability– Repeatability– Reproducibility
User-Centered Design www.user-centereddesign.com41
![Page 42: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/42.jpg)
Humans are complex processing systems, dynamic and highly
affected by context, and all different
User-Centered Design www.user-centereddesign.com42
![Page 43: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/43.jpg)
Perceptual issues(our brain’s versus our senses)
…
User-Centered Design www.user-centereddesign.com43
![Page 44: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/44.jpg)
+ •
44
![Page 45: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/45.jpg)
+
45
![Page 46: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/46.jpg)
46
![Page 47: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/47.jpg)
47
![Page 48: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/48.jpg)
48
![Page 49: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/49.jpg)
49
![Page 50: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/50.jpg)
50
Müller-Lyer Illusion
![Page 51: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/51.jpg)
51 User-Centered Design www.user-centereddesign.com
![Page 52: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/52.jpg)
52
![Page 53: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/53.jpg)
53
![Page 54: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/54.jpg)
54
![Page 55: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/55.jpg)
55 User-Centered Design www.user-centereddesign.com
![Page 56: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/56.jpg)
56 User-Centered Design www.user-centereddesign.com
![Page 58: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/58.jpg)
The Gestalt Principles
User-Centered Design www.user-centereddesign.com58
![Page 59: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/59.jpg)
Proximity
59
![Page 60: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/60.jpg)
Similarity
60
![Page 61: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/61.jpg)
Parallelism (Similarity)
Elements that are parallel to each other appear more related than elements not parallel to each other.
User-Centered Designwww.user-centereddesign.com
![Page 62: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/62.jpg)
Symmetry
The idea that when we perceive objects we tend to perceive them as symmetrical shapes that form around their center (competes with proximity).
User-Centered Designwww.user-centereddesign.com
![Page 63: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/63.jpg)
Continuation
63
![Page 64: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/64.jpg)
Common Region
Elements tend to be grouped together if they are located within the same closed region.
![Page 65: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/65.jpg)
Prägnanz (German for “pithiness”)
User-Centered Designwww.user-centereddesign.com
65
![Page 66: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/66.jpg)
Common Fate (Synchrony)
Elements moving in the same direction are perceived as being more related than elements that are stationary or that move in different directions. Elements that change at the same time group together.
User-Centered Designwww.user-centereddesign.com
![Page 67: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/67.jpg)
Unity
67
![Page 68: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/68.jpg)
Emergence
Emergence is the process of complex pattern formation from simpler rules
User-Centered Designwww.user-centereddesign.com
![Page 69: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/69.jpg)
Reification
the constructive or generative aspect of perception
User-Centered Designwww.user-centereddesign.com
![Page 70: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/70.jpg)
Invariance
simple geometrical objects are recognized independent of rotation, translation, and scale
User-Centered Designwww.user-centereddesign.com
![Page 71: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/71.jpg)
Multistability
the tendency of ambiguous perceptual experiences to pop back and forth unstably between two or more alternative interpretations
User-Centered Designwww.user-centereddesign.com
Necker Cube Rubin Cube
![Page 72: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/72.jpg)
Closure
72
![Page 73: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/73.jpg)
73
![Page 74: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/74.jpg)
74
:-)
:-(
;-)
8-)
![Page 75: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/75.jpg)
![Page 76: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/76.jpg)
Closure Example 2
76
![Page 77: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/77.jpg)
Closure Example 3
77
![Page 78: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/78.jpg)
Closure Example 4
78
![Page 79: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/79.jpg)
79
![Page 80: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/80.jpg)
80
![Page 81: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/81.jpg)
…our perceptual abilities are limited in the presence
of noise…
User-Centered Design www.user-centereddesign.com81
![Page 82: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/82.jpg)
82 User-Centered Design www.user-centereddesign.com
THE QUICK BROWN FOX JUMPED OVER THE LAZY DOG’S
BACK.
82
![Page 83: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/83.jpg)
83 User-Centered Design www.user-centereddesign.com
THE QUICK BROWN FOX JUMPED OVER THE LAZY DOG’S
BACK.
83
![Page 84: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/84.jpg)
84 User-Centered Design www.user-centereddesign.com
THE QUICK BROWN FOX JUMPED OVER THE LAZY DOG’S
BACK.
84
![Page 85: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/85.jpg)
85 User-Centered Design www.user-centereddesign.com
The quick brown fox jumped over the lazy dog’s
back.
85
![Page 86: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/86.jpg)
…our presumptions effects what and how we interpret
things…
User-Centered Design www.user-centereddesign.com86
![Page 87: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/87.jpg)
Jack and Jill wentwent up the
Hill to fetch aa pail of milk
87
![Page 88: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/88.jpg)
88
![Page 89: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/89.jpg)
89
![Page 90: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/90.jpg)
90
![Page 91: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/91.jpg)
91
![Page 92: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/92.jpg)
92
![Page 93: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/93.jpg)
…our attention and cognitive abilities are
limited, easily manipulated, and
specialized…
User-Centered Design www.user-centereddesign.com93
![Page 94: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/94.jpg)
FINISHED FILES ARE THE RE-SULT OF YEARS OF SCIENTIF-
IC STUDY COMBINED WITH THE EXPERIENCE OF MANY YEARS
94
![Page 95: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/95.jpg)
95
![Page 96: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/96.jpg)
RedGreenBlue
OrangeYellowBlack96
![Page 97: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/97.jpg)
StroopStroopStroopStroopStroopStroop97
![Page 98: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/98.jpg)
OrangeYellowGreenBlackBlueRed98
![Page 99: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/99.jpg)
User-Centered Design www.user-centereddesign.com
Test Your Attention
99
![Page 100: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/100.jpg)
User-Centered Design www.user-centereddesign.com
Change Blindness(The Person Swap Experiment)
100
![Page 101: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/101.jpg)
User-Centered Design www.user-centereddesign.com101
Test your skill
![Page 102: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/102.jpg)
…our psychological state (e.g., anxiety) affects our
performance…
User-Centered Design www.user-centereddesign.com102
![Page 103: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/103.jpg)
Use of Confidence Intervals
When working with small samples, confidence interval provide a way to represent uncertainty in test results
Since each sample and each test is different, the confidence level tells the informed reader the likelihood that another sample will provide the same results. (In other words, if you ran the test again, what value are you likely to get next time?)
Typical confidence intervals in research include the 90% or 95% confidence interval. Behavioural research often uses a 80% confidence interval.
User-Centered Design www.user-centereddesign.com103
![Page 104: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/104.jpg)
Use of Confidence Intervals (continued) “You just finished a usability test. You had 5
participants attempt a task in a new version of your software. All 5 out of 5 participants completed the task. You rush excitedly to tell your manager the results so you can communicate this success to the development team. Your manager asks, ‘OK, this is great with 5 users, but what are the chances that 50 or 1000 will have a 100% completion rate?’ ”- Jeff Sauro
The confidence level tells the informed reader the likelihood that another sample will provide the same results. In other words, if you ran the test again, what value are you likely to get next time?
User-Centered Design www.user-centereddesign.com104
![Page 105: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/105.jpg)
Use of Confidence Intervals (concluded) Confidence intervals when testing with 8 people range
from 37% (0 out of 8 or 8 out of 8) to between 50%-70% (all other values)
For example, if 6 out of 8 people successfully completed a task in your test, you can only predict that somewhere between 20% and 97% of all people would complete the task (assuming all conditions of validity and reliability have been met)
If you want to confidently state, based on your testing, that 9 out of 10 people will be able to successfully complete a task, and all conditions of validity and reliability have been met, you need to test 430 people and 400 of them have to successfully complete the task
User-Centered Design www.user-centereddesign.com105
![Page 106: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/106.jpg)
“Observational” Techniques
User-Centered Design www.user-centereddesign.com106
![Page 107: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/107.jpg)
107
Contextual Inquiry
Field Study– Sometimes (incorrectly) called “ethnography”
Direct observation of – intended users – performing the intended tasks– in the intended environment
(Should be) non disruptive, so its limited in its ability to be diagnostic or exploratory
Common functions are viewed– Incomplete view of a system
Can be time consuming and logistically prohibitive Best for directly observable data from a “safe”
distance
User-Centered Design www.user-centereddesign.com
![Page 108: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/108.jpg)
108
Performance-based Testing
Must be non-disruptive to be valid– Need a fully operational system, mock up, or prototype– In context (ideally not in a lab)
Need objective measures Need large enough sample to avoid the need for
statistical analysis Applicability in (some) web-based situation
– sometimes called “Un-moderated Remote Usability Testing”
Examples– Redundant High Centered Tail Lights– Google
However…– Limited ability to to determine cause– Limited ability to determine possible
changes/improvementsUser-Centered Design www.user-centereddesign.com
![Page 109: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/109.jpg)
109
Think Aloud Protocol
Most widely used (which is not a good thing) Highly disruptive to performance No reliable evidence of its efficacy When used on existing systems or interactive
prototypes/mockups– Issues of the ability for users to be introspective– Issues of distraction (split attention)– Issues of verbal overshadowing– Issues of increased anxiety– Issues of projected responding
Suitability for concept presentation and cognitive walkthroughs on non-operational products (e.g., story boards, static screen flows, Wizard of Oz walkthroughs)
User-Centered Design www.user-centereddesign.com
![Page 110: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/110.jpg)
110
Interrupted Task-Based Testing
A compromise approach that allows for exploration of issues without being overly disruptive when issues are not present
Can be used for exploratory testing on an existing design
Can be used for exploring possible design alternatives
Still has threats to validity and reliability– Highly Disruptive– Telegraphing– Audience Effect (Hawthorne Effect)– Projected Responding
User-Centered Design www.user-centereddesign.com
![Page 111: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/111.jpg)
How toDesign & Conduct
anInterrupted Task-
based Test
User-Centered Design www.user-centereddesign.com111
![Page 112: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/112.jpg)
The Rules
Testing should be developed using the model of a scientific experiment
Testing should follow the ethical guidelines for the treatment of human subjects– Informed consent– Confidentiality
Testing should not be hampered by trying to support statistical analysis
User-Centered Design www.user-centereddesign.com112
![Page 113: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/113.jpg)
113
Test Set-up
What’s the hypothesis?– Required for research– Required for usability testing?
Define Your Variables– Dependent and Independent Variables– Confounding Variables– Operationalize Your Variables
User-Centered Design www.user-centereddesign.com
![Page 114: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/114.jpg)
114
Participant Issues
User-types– Users versus user surrogates– All profiles or specific user profiles/personas?– Critical segments?
How many?– Relationship to statistical significance– “Discount Usability” – who’s rule?– No less then 3 from any group
Participant stipends Over recruiting Scheduling
User-Centered Design www.user-centereddesign.com
![Page 115: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/115.jpg)
115
Test Set-up
Select a Protocol– Within versus Between Subject Designs– Based on time commitment & number of
designs/products– Practically: Try for an unbalanced within subject
design Selecting a Format
– Think aloud– Interrupted Task-based– Performance-based (with or without a critical
incidence analysis) Special Case(s)
– Co-discovery– “Group”
User-Centered Design www.user-centereddesign.com
![Page 116: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/116.jpg)
116
Defining Task Scenarios
Scenarios are contrived for testing, may not be representative of real world usage patterns, and are NOT always required
Short, unambiguous tasks to explore areas of concern, redesign, or of interest
Wording is critical– In the user’s own terms– Does not contain “seeds” to the correct
solution Enough to form a complete test but able to stay
within the time limit– Flexibility is key– Variations ARE allowed
User-Centered Design www.user-centereddesign.com
![Page 117: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/117.jpg)
117
Preparing Test Materials
Consent form! Video release form! Receipt and confidentiality agreement! Demographic survey Facilitator’s Guide
– Introductory comments– Participant task descriptions– Questionnaires, SUS, Cooper-Harper, etc.
Note Taker’s Forms
User-Centered Design www.user-centereddesign.com
![Page 118: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/118.jpg)
118
Piloting the Design
Getting subjects– Convenience sampling– Cells and Power
Collect data Check task wording Check timing
User-Centered Design www.user-centereddesign.com
![Page 119: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/119.jpg)
Facilitating
Rogerian principles apply– Unconditional Positive Regard– Empathy– Congruence
Rogerian techniques are used– Minimal encouragers– Reflections– Summarization– Open ended questions
Objectiveness
User-Centered Design www.user-centereddesign.com119
![Page 120: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/120.jpg)
120
Collecting Data
Collecting data– The data is NOT in the interface, the data is in the
user!– Remote testing?
Collecting observed data– Behavior– Reactions
Collecting participant comments Collecting subjective data
– Pre-test data– Post-scenario data– Post-test data
User-Centered Design www.user-centereddesign.com
![Page 121: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/121.jpg)
Reporting Results
User-Centered Design www.user-centereddesign.com121
![Page 122: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/122.jpg)
User-Centered Design www.user-centereddesign.com122
![Page 123: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/123.jpg)
User-Centered Design www.user-centereddesign.com
Efficiency Data – Time on Task
Efficiency data can be operationalized in number of ways – time on task being the most common
Time on task can be measured objectivelyExternal time is important to management,
but is not necessarily important users and time on task does not correlate with effectiveness (except in extreme cases)
123
![Page 124: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/124.jpg)
Sample ToT Data – Controlled Experiment*
User-Centered Design www.user-centereddesign.com
150
250
350
450
550
650
750
850
950
1050
1150
1250
1350
0
1
2
3
4
5
6
7
8
9
10
System A: ToT Time in Seconds
Num
ber
of
Indiv
iduals
150
250
350
450
550
650
750
850
950
1050
1150
1250
1350
0
1
2
3
4
5
6
7
8
9
10
System B: ToT Time in Seconds
Num
ber
of
Indiv
iduals
*Source: UCD, Inc. – Voting System Usability Compliance Test Development Report for NIST
124
![Page 125: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/125.jpg)
User-Centered Design www.user-centereddesign.com
Efficiency Data – Other Measures
The following measures have been proposed– Number of clicks– Number of pages– Number of errors– Number of times the back button is used– “Pogo sticking”
There is no construct validity for any of these measures against task performance
Though the data is not necessarily normally distributed, with fairly large samples, a t-test can be used to complete time provided the test as validity– It can be invalidated by sampling issues– It is not valid in think aloud or exploratory protocols
(the types used to understand what is wrong and to attempt to find solutions)
125
![Page 126: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/126.jpg)
User-Centered Design www.user-centereddesign.com
Satisfaction Data
Satisfaction data can be operationalized in a number of ways, but is always opinion data– Standardized survey instrument (e.g. SUS, SUMI,
QUIS)– Simple Likert item and Likert scale assessments
Satisfaction data suffer from numerous issues that threaten their validity– Halo effect– Leniency effect– Strictness effect– Projected responding– Bias– Usability Issues (agreed understanding)
Satisfaction data does not correlate with performance
126
![Page 127: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/127.jpg)
Post Test Analysis of Approx. 3000 Sessions*
User-Centered Design www.user-centereddesign.com
*Source: Jeff Sauro, Measuring Usability
Subjective Ease of Use Assessment (when successful)
127
![Page 128: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/128.jpg)
Post Test Analysis of Approx. 3000 Sessions*
User-Centered Design www.user-centereddesign.com
Subjective Ease of Use Assessment (when unsuccessful)
128
![Page 129: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/129.jpg)
Effectiveness Data
Effectiveness data can be operationalized in a number of ways but is generally operationalized as success or failure to complete a task
Completion rate as a pass/fail criteria can be measured objectively if the criteria is pre-determined and is not subjective
Best estimates, error rate, and the confidence interval can be calculated easily for pass/fail measure of completion rate using a Binomial calculation
User-Centered Design www.user-centereddesign.com129
![Page 130: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/130.jpg)
Descriptive Statistics
But the data often shows other patterns such as bimodal distributions. In these cases, the average and standard deviation are not adequate…
User-Centered Design www.user-centereddesign.com130
![Page 131: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/131.jpg)
User Ratings
User-Centered Design www.user-centereddesign.com131
Score
Num
ber
who
got
th
at s
core
![Page 132: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/132.jpg)
Correlated User Ratings
User-Centered Design www.user-centereddesign.com
SUS
Cooper Harper
132
![Page 133: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/133.jpg)
User-Centered Design www.user-centereddesign.com133
Findings from Sets of User Ratings
DC – MCH DataMemphis – MCH Data
Memphis – SUS Data DC – SUS Data
![Page 134: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/134.jpg)
Reportable “Results”
Violations of industry standards and best practices are reportable results from testing (though many should have been included in any expert review prior to testing)
Direct user comments may or may not be reportable, based on the observers assessment of the comment
Direct user behaviour is generally reportable, but only if confirmed to be behaviour based on a design issue and/or behaviour that is consistent throughout testing
An observation of a reaction suggestive of a cognitive issues, regardless of its effect on observable behaviour, is reportable provided there is a basis for that assumption
Behaviours that did not occur in testing but are suspected to occur under different conditions are reportable provided they re based on prior experience and there is a basis for that behaviour
Subjective data is reportable to support other findings, but this support may be inversely correlated with observation or performance
User-Centered Design www.user-centereddesign.com134
![Page 135: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/135.jpg)
Direct Observation or Comment
User-Centered Design www.user-centereddesign.com
Participants had difficulty understanding what content was
searched.
Many thought all content in Clinical
Trials would be searched, not just
ongoing trials A few participants wanted to use the global NCI search
to search Clinical Trials (consider labelling this “Search NCI” or “NCI
Search”
Some participants responded to the term “Find” even when the search form was on the
page.
![Page 136: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/136.jpg)
Design Guidelines
All navigation should be in grouped
together.
All navigation should be in grouped
together.
![Page 137: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/137.jpg)
Prior Research Findings
User-Centered Design www.user-centereddesign.com
Bold form labels draws users eyes away from the form and reduces
usability. Consider removing the bold and
possibly bolding the content.
137
![Page 138: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/138.jpg)
138
Knowledge of Human Perception:
User-Centered Design www.user-centereddesign.com
There are 50
hyper links on the
home page (not including
primary nav.)
representing four levels
within the clinical trial
section and direct links to
other parts of NCI
![Page 139: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/139.jpg)
Industry Standards and Best Practices
Participants (without prior exposure) failed to recognized the five primary disciplines as navigational elements. The most common expectation (if noticed at all) was that the links would provide definitions of the terms.
User-Centered Design www.user-centereddesign.com139
![Page 140: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/140.jpg)
Conclusions
Any testing is better than no testing
The more you know about experimental design the better your testing will be, but the more you know about users the better the data you can get from any testing
Unless you have valid and reliable, large sample data available, the is little distinction between a true expert review and user-based testing, but experts need users to “see” the data
User-Centered Design www.user-centereddesign.com140
![Page 141: An Introduction to Usability Testing Bill Killam, MA CHFP Adjunct Professor University of Maryland bkillam@user-centereddesign.com User-Centered Design](https://reader038.vdocument.in/reader038/viewer/2022110319/56649c725503460f9492481a/html5/thumbnails/141.jpg)
Conclusions (concluded ;-) )
Testing is best done early and often as part of the design process
The intent of testing should be to not just to know what happened, but to determine why it happened and to figure out what, if anything, can be done about it
Testing with human subject is highly valuable, can be deeply insightful, but is serious business and should not be conducted casually
User-Centered Design www.user-centereddesign.com141