usability testing - hci lecture
TRANSCRIPT
![Page 1: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/1.jpg)
Usability Testing
The following content is licensed under a Creative Commons Attribution 4.0 International license (CC BY-SA 4.0) Valentin Schwind1
Image Source: https://pxhere.com/de/photo/205682
![Page 2: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/2.jpg)
Learning Goals
▪ Understanding the principles of usability testing and user experience sampling
▪ Understanding when and how usability tests can be conducted
▪ Learn strengths and weaknesses
Usability Testing Valentin Schwind2
![Page 3: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/3.jpg)
Basics
▪ Testing a prototype with a set of representative users
▪ For some tests, users must have certain domain, product and application-specific knowledge and experience
▪ Users complete typical tasks with:
▪ a prototype
▪ a part of the system
▪ the complete system
▪ with competitive systems
▪ Can be conducted:
▪ in early in the design processes to find problems to improve the product (formative evaluation)
▪ to validate the design against specific goals (summative evaluation)
Usability Testing Valentin Schwind3
![Page 4: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/4.jpg)
Benchmark Testing
▪ Task-based Usability Tests
▪ Ability to complete tasks
▪ The time it takes to complete tasks or find information
▪ Ability to support users (e.g. learning, reducing errors,…)
▪ Experienced-based Usability Tests
▪ Attitudes toward visual appearance
▪ Attitudes toward trust and credibility
▪ Perceptions of ease, usefulness, and satisfaction
▪ Emotions, immersion, playability (e.g. games)
▪ Visibility of an interface
Usability Testing Valentin Schwind4
![Page 5: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/5.jpg)
Wizard of Oz
▪ User-based evaluation of unimplemented technology
▪ Unknown to the user that a human or team is simulating some or particular functionality of the system
▪ Examples
▪ speech and natural language applications
▪ command languages
▪ imaging systems
▪ pervasive computing applications
▪ artificial intelligence
Valentin Schwind5Usability Testing
Image Source: https://pxhere.com/cs/photo/1548563
![Page 6: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/6.jpg)
Wizard of Oz Setup
Usability Testing Valentin Schwind6
J. F. Kelley. 1983. An empirical methodology for writing user-friendly natural language computer applications. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’83).
User Experimenter („Wizard“)
![Page 7: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/7.jpg)
Test Locations
▪ Lab Evaluations
Usability Testing Valentin Schwind7
Images from: https://pxhere.com ID: 1599048, 718703, 802607, 700764
Lab In Situ
Remote Crowd
![Page 8: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/8.jpg)
Interviews
▪ Unstructured
▪ Not directed by a script
▪ Rich but not replicable
▪ Structured
▪ Tightly scripted, often like a questionnaire
▪ Replicable but may lack richness
▪ Semi-structured
▪ Guided by a script but free to explore interesting issues in more depth
▪ Good balance between richness and replicability
Usability Testing Valentin Schwind8
![Page 9: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/9.jpg)
How to Ask Questions
▪ Clear and simple, not too broad
▪ “How do you like the UI?” is too general!
▪ Affording logical, sometimes quantitative answers
▪ Bad questions give unusable or wrong answers
▪ Open (W-questions) vs. closed questions
▪ Users don’t always answer truthfully
▪ Lack of knowledge, bad estimates, embarrassment
▪ So formulate questions carefully, maybe indirectly
▪ No leading questions!
▪ For initial input, do not focus on presenting your design ideas, but on learning about the task
Usability Testing Valentin Schwind9
![Page 10: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/10.jpg)
How to Ask Questions
▪ Clear and simple, not too broad
▪ “How do you like the UI?” is too general!
▪ Affording logical, sometimes quantitative answers
▪ Bad questions give unusable or wrong answers
▪ Open (W-questions) vs. closed questions
▪ Users don’t always answer truthfully
▪ Lack of knowledge, bad estimates, embarrassment
▪ So formulate questions carefully, maybe indirectly
▪ No leading questions!
▪ For initial input, do not focus on presenting your design ideas, but on learning about the task
Usability Testing Valentin Schwind10
![Page 11: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/11.jpg)
Running an Interview
1. Introduction: Introduce yourself, explain the goals of the interview, reassure about the ethical issues, ask to record, present any informed consent form
2. Warm-up: Make first questions easy and non-threatening
3. Main body: Present questions in a logical order
4. A cool-off period: Include a few easy questions to defuse tension at the end
5. Closure: Thank interviewee, signal the end, e.g., switch recorder off
▪ Enriching the Interview: Present prototypes or devices forprompting interviewee
▪ Data Recording: Notes, audio, video, photographs
Usability Testing Valentin Schwind11
![Page 12: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/12.jpg)
Think Aloud Protocol
▪ Users are asked to tell what they are
▪ doing
▪ looking at
▪ wondering about
▪ feeling
▪ thinking
▪ ….while performing specific tasks during a Usability Test
▪ Experimenter record statements without commenting or answering questions
▪ Goal: reveal thoughts of the users to be able to relate to why they act the way they do
Usability Testing Valentin Schwind12
![Page 13: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/13.jpg)
Live Video Observation
1. Screen, subject, environment…
2. Computer for the test user,
▪ run application to test
▪ export the screen (e.g. VNC)
3. Computer for the observer
▪ see the screen from the subject
▪ attach 2 web cams
▪ text input for observer notes
▪ capture this screen
Example
Usability Testing Valentin Schwind13
Subjects screen
Cam1
Editor
Cam2
time
Subjects screen
Test system
Observer system
![Page 14: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/14.jpg)
Observations
▪ Moderated/Supervised, with the evaluator observing theparticipant in real time
▪ Screen-sharing/webcam: Adobe Connect, GoTomeeting, Zoom, NetMeeting, LiveLook, UserVue, Skupe, WebEx, Glance, Youguu
▪ Unmoderated/Unsupervised, with the participant workingwithout direct observation or interaction
▪ Live sites/apps: UserZoom, RelevantView, WebEffective, Webnographer (qualitative), ClickTale, ClickHeat (quantiative)
▪ Testing conceptual artifacts: online card sorting, OptimalSort, WebSort
▪ Surveys
Valentin Schwind14
![Page 15: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/15.jpg)
General Procedure
▪ Test planning
▪ Guideline-based, task plan, information material
▪ Stakeholder meetings / interviews / participatory design
▪ Pretest or pilot → inviting participants
▪ Test sessions
▪ Experimenter interviews, watches, listen, take notes
▪ Experimenter intervenes when users are clearly in trouble
▪ Typically recorded either by video and/or automated testing tool
▪ Post-test or debrief
▪ Analysis, interpretation, and presentation of the results.
▪ Collects ideas for potential improvements
Usability TestingValentin Schwind15
![Page 16: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/16.jpg)
Usability Testing Valentin Schwind16
Image from https://www.testingtime.com/blog/usability-test/
![Page 17: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/17.jpg)
Number of Users
▪ The number of usability problems found in a usability test:
𝑃𝑟𝑜𝑏𝑙𝑒𝑚𝑠𝐹𝑜𝑢𝑛𝑑(𝑖) = 𝑁 (1 − (1 − 𝐿 ) 𝑛 )
→ “To find 85% of the usability problems, you need five users.”
Usability Testing Valentin Schwind17
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
![Page 18: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/18.jpg)
Feedback Analysis
▪ Summarize results
▪ Transcribe the notes and statements of participants
▪ Effectiveness (e.g. task success, number of errors)
▪ Efficiency (e.g. task completion time, interaction path, need for assistance)
▪ Group by usability issue
▪ Value usability issues
▪ Frequency
▪ Severity (e.g. caused mistakes, frustration)
▪ Derive concrete design ideas to improve the prototype
▪ Questionnaires
Usability Testing Valentin Schwind18
![Page 19: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/19.jpg)
Identifying Usability Issues
Valentin Schwind19
ID Task Criticality Where Description Impact P1 P2 P3 # Severity
1 Login 3 Login Page Didn‘t recognize the
Facebook option
5 1 0.33 4.95
2 Creating a
post
4 Blog Took too long to load
text input page
2 1 1 0.66 5,28
3 Creating a
post
5 Blog Struggled to find the
button
3 1 1 0.66 9,9
4 Login 3 Registration No (auto) login after
creating new account
4 1 1 1 1.0 12
How important is the task
affected by this issue?
The sum of issues for
P1+P2+P3
Severity = Criticality x Impact x
Frequency
![Page 20: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/20.jpg)
Further Approaches
▪ Contextual Inquiries: researcher watches the user's normal activities and, then, discusses those activities with the user
▪ Longitudinal Studies
▪ Experience Sampling: participants report on their thoughts, feelings, behaviors, and/or environment on multiple occasions over time
▪ Diary Studies: participants record entries about their everyday lives in a log, diary or journal about the activity or experiences
▪ Focus Groups: a heterogonous group of people, whose reactions are studied in guided discussions
▪ Brainstorming: generating and evaluating ideas
Valentin Schwind20
![Page 21: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/21.jpg)
Further Approaches
▪ Card sorting/Affinity diagramming: discovering the latent structure in an unsorted list of statements or data
▪ Context of Use/Scenario of use/Task analysis: collecting and analyzing information about potential users, tasks, and constraints
▪ Workshops: user and developer identify usability requirements that can be tested in the development process
Valentin Schwind21
![Page 22: Usability Testing - HCI Lecture](https://reader034.vdocument.in/reader034/viewer/2022042204/6258afd2e12ccf7dbd2bcf80/html5/thumbnails/22.jpg)
References
▪ Dumas, J.S. & Redish, J.C. (1999). A practical guide
to usability testing. Exeter: intellect.
▪ Tullis, T. & Albert, B. (2008). Measuring the user experience.
San Francisco: Morgan Kaufmann.
▪ Joseph S. Dumas (2002). User-based evaluations, The
human-computer interaction handbook: fundamentals,
evolving technologies and emerging applications. Mahwah:
Lawrence Erlbaum Associates, Inc.
▪ David W. Martin (2003). Doing Psychology Experiments.
Belmont: Wadsworth Publishing.
▪ https://measuringu.com/
Valentin Schwind22Usability Testing