user testing and modeling
DESCRIPTION
User Testing and Modeling. ICS 205 Chris Wesson Christina Wuerth November 14, 2003. User Testing. What is User Testing Measuring performance of typical users doing typical tasks Goal: - PowerPoint PPT PresentationTRANSCRIPT
User Testing and Modeling
ICS 205
Chris Wesson
Christina Wuerth
November 14, 2003
User Testing
What is User Testing– Measuring performance of typical users doing
typical tasks
Goal:– Obtain objective performance data showing how
usable a system is in terms of usability goals (ease of use, learnability)
User Testing
Applied form of experimentation Tests whether product is usable by intended
user group(s) Measures/Records:
– Time to complete typical tasks– Number and type of errors– Routes users take through tasks (Web)
Typically 6-12 users
User Testing
Is part of Usability Testing:– Obsevations– Questionaires– Interviews– User Testing
User Testing and Modeling
Doing User Testing
Experiments
Predictive Models
Keys to Doing User Testing
Control testing conditions Careful planning Same conditions for each participant Measurements correspond to test Explicit assumptions
DECIDE framework
Q: How Long Should User Testing Process Take?
A: As little time as possible while still yielding useful information.
On average most companies take more than
8 weeks for user testing
Testing of small piece of well developed procedures can be done in less than 1 week if needed
Things to Consider in Planning a User Test
What aspects of the product need to be improved? Do the participants in the study represent actual
users of the product? What tasks should the participants perform? What information will you observe during the tests? How will you analyze the data you collect? What will you do with this information once it is
analyzed?
User Test Planning Steps
1. Define goals and concerns2. Decide who will participate3. Recruit participants4. Select and organize tasks to be tested5. Create task scenarios6. Decide how to measure usability7. Prepare other materials for the tests (questionnaires, etc.)8. Prepare the testing environment9. Prepare the test team10. Conduct pilot tests making changes where needed
The Testing Team
The Roles:Usability Specialists
The usability specialists know what can be accomplished in a user tests and how a test should be planned. They should also understand the interface and potential problems users may have with it.
Designers/DevelopersThe designers and developers understand the product and
what the user should be able to do with it.
Technical CommunicatorsTechnical communicators know what aspects of the
documentation should be tested and also see potential problems in how the product communicates to the users.
More User Test Team Roles
TrainersThe trainers understand the problems that users will have when learning to use the product.
Marketing SpecialistsMarketing specialists know who the users will be.
Helpdesk/Customer Service RepsHelpdesk and customer service reps know what problems users have had with earlier versions of the software.
Preparing for the Test
1. Schedule activities2. Assign roles and responsibilities3. Train team4. Write out a test plan5. Practice by running pilot tests
Throughout the entire process remember to:- track progress - document all decisions
DECIDE:User Testing Guideline
D etermine goals E xplore questions C hoose paradigm and techniques I dentify practical issues D eal with ethical issues E valuate, analyze, present data
Determine Goals, Explore Questions
User testing is best used for testing prototypes and working systems
Goals can be broad (how usable?)
Specific questions required to focus study (can task X be done in time Y?)
Choose Paradigm and Techniques
User testing is part of Usability testing paradigm
Data can be recorded using:– Video– Interaction log– User satisfaction questionnaires– Interviews
Identify Practical Issues:Design Typical Tasks
Create set of “completion” tasks (finding a website)
Choosing which tasks to test is critical Tasks generally last 5-20 minutes Tasks are often straightforward Complex tasks (create design, solve
problem) are okay Start simple to build user confidence
Identify Practical Issues:Select Typical Users
Must know users’ characteristics
Most important: previous experience
Use short questionnaire to identify users
Equal number of males and females
Identify Practical Issues:Prepare Testing Conditions
Control testing environment
Minimize outside influences/noises that could distort results
Identify Practical Issues:Plan How to Run Tests
Create schedule and scripts for running tests Start with familiarization task Contingency plan for spending too much time
on a given task Avoid long tasks and long testing procedure Session < 1 hour
Deal with Ethical Issues
Informed consent form
Point out any:– One-way mirrors– Video cameras– Interaction logging
Evaluate, Analyze,and Present Data
Performance measures recorded from video, interaction logs
Only use simple statistics:– Maximum Minimum– Mean Standard Deviation
Allows evaluators to compare performance between systems and across tasks
Establishing Goals and Concerns
Step 1 of the planning phase is to define goals and concerns.
Q: What is a goal?
A: A quantitative usability aim that can be stated as a declarative sentence
example: A user should be able to locate the print menu item in less than 5 seconds with less than 3 errors during first attempt.
Q: What is a concern?
A: A concerns are questions raised about the usability of the product – these are often raised while planning for the tests.
example: While a user find the new error messages confusing?
Establish Which Goals and Concerns will be tested.
Not all goals and concerns can be tested simultaneously.
Example of 2 conflicting concerns:1. Will the user use feature X?2. Will the user find X easy to use?
These two questions can’t be answered at the same time instead you must prioritize.
Using Concerns to Plan the User Test
Use General concerns to decide who your participants should be
Example: Will new users find this product easy to learn?Make sure you test with some beginning users.
Use specific concerns to decide which tasks should be performed.
Example: Will users be able to find the print command quickly? In less than 5
seconds? Have the users perform a task of printing.
Sources of Goals and Concerns
1. Task Analysis and Quantitative usability goals
Example:
General: Menus should be easy to navigate
Quantitative Goal: Users should find menu choice in less than 2 minutes with no more than 2 wrong choices when first need it. Task should be done with no errors in less than 1 minute after first attempt.
Sources of Goals and Concerns
2. Timely Issues – Certain concerns arrive at different stages of development and should be resolved at that time. Example:
1) Two different design philosophies that you can’t decide between…plan a user test at start of the project to test prototypes of each
2) There has been previous testing earlier in the project, now you want to test a new feature before continuing development.
Sources of Goals and Concerns
3. Heuristic Analysis and Expert Reviews a) Problems that have been predicted by heuristic analysis and
expert reviews should be candidates for user tests.
b) Problems predicted by designers, planners, developers, human factor specialists, technical communicators should be tested.
c) Concerns found by reflecting on the product itself should be tested.
Sources of Goals and Concerns
4. Previous Tests Example:
iterative testing with rapid prototypes
run several user tests with the same concerns
Example:
large scale user tests that raise future concerns
retest with same concerns to make sure changes improved the problem
Who Should be Participants?
Participants must be like the people who will actually use the product.
Develop a user profile then choose participants that fit the profile.
Developing a User Profile
Base profile on:
General market research
Analysis of customers of competitors’ products
Focus group sessions
Observing and interviewing prospective users
Developing User Profile
1. Think about relevant characteristicsa) those that all users shareb) those that might make a difference among users
example: shared characteristic:
users will be undergraduate university studentsdifferences:
users will have lots of computer experienceusers will have very little computer experienceusers will have registered for classes beforeusers will have never registered for a class before
Developing User Profile
2. Decide which characteristics matter most in establishing usability
experience and motivation contribute more towards and individual’s understanding than education, income, age, etc.
Good Factors to Consider1. Work experience2. General computer experience3. Specific computer experience4. Experience with this product5. Experience with similar products
Developing User Profile
3. Think broadly about who the users will beSample questions to reflect on:
1. Not just those currently working, but what about new hires? What is the rate of turnover that the software will need to handle?
2. Don’t limit yourself to the current market…what about growth, who might want to use this product in the future?
3. What other areas of the corporation may what to adopt this software?
4. Reflect on differences within a category of users. Older users versus younger users? Users who have done the job longer versus users that more adaptable to computer environments?
Developing a User Profile
1. Product Name: Class Registration Software
2. General Characteristics of User Population
University Students
3. Characteristics of Users Relevant to the test
Year in School
Computer Experience
Previous Experience Registering for Classes
Developing a User Profile
4. Which Characteristics listed in 3 should all users in the test have in common & how will you define them?All users should be students currently enrolled in a universityThere will undergraduate and graduate students
Developing a User Profile
5. Which characteristics listed in 3 will vary in the test and how will you define them?
Computer Experience will vary.
Novice = < 1 year experience
Medium = > 1 year experience used < 5 days a week regularly
Expert = > 1 year experience used > 5 days a week regularly
Select Subgroups for a User Test
Subgroups are people who share specific characteristics important to the user profile
Divide groups by one characteristic at a time.
Example:University Students
Subgroup 1 Subgroup 2Undergraduates Graduates
Subgroup 1a Subgroup 2aUndergraduates GraduatesLittle computer experience Little computer experience
Subgroup 1b Subgroup 2bUndergraduates GraduatesLots of computer experience Lots of computer experience
Define Characteristics for Each Subgroup
Define what you mean by characteristicsExample
novice = less than 1 year experience with this product
medium = 1 year to 3 years experience with this product
expert = more than 3 years experience with this product
Qualify Characteristics of the Subgroups
1. The middle group is often omitted from the subgroups in user studies since most problems found here will be found in novice and expert studies
2. Select a range of participants in each subgroup.Example: Subgroup of novices with < 1 year experience
Find users with 1 month, 6 months, and 11 months experience
3. Establish range of each subgroupset min and max for subgroup participantsExample: Subgroup of experts with > 3 year experience
min would be 3 years but perhaps you want max experience to be 5 years…you don’t want someone whose a hacker even though they may have lots of experience
How Many Participants?
Depends on:1. How many subgroups you need2. Time and money constraints3. How important it is to compute statistically
significant results
* Most of the time all you will care about is inferential statistics
How Many Participants?
6 – 12 participants is the typical choice
With a minimum of 3 individuals in each subgroup.
* Problems will show up across subgroups
Making the Most of a limited Number of Participants
1. Decide which characteristics are the most important so that your subgroups will be most useful
2. Collect relevant information from participants via interviews or questionnaires to help account for differences that show up in results.Example: one novice user performs faster than all the others, but you find out that he has used a similar product before
3. Select people in subgroups representing the full range of the group
Experiments
User testing based on scientific experimentation
Aim: test hypothesis that predicts a relationship between variables
Variables
Independent: manipulated by researcher Often multiple independent variables:
– System version, User experience
Dependent: affected by independent variable Common dependent variable:
– Time to complete task, Number of errors
Assigning Participants to Conditions
Three Experimental Designs:
Different participants for all conditions Same participants for all conditions Matched pairs of participants
Matched Participants
Participants matched in pairs
Based on specific characteristics (expertise, gender, etc.)
Used when participants cannot perform in all conditions
Design Comparison
Design Advantages Disadvantages
Different Participants
No order effects. Many participants needed. Individual differences can be a problem.
Same Participants
Eliminates individual differences.
Must counterbalance ordering effects.
Matched Participants
No ordering effects. Reduces effects of individual differences.
No guarantee subjects are match across all variables.
Data Collection
Data should measure users’ performance
Typical measures:– Response times– Number of errors– Time to complete task
Data Analysis
Questions to ask:– Do data sets from two conditions look similar or
different?– Any extreme atypical values?– If so, what does that mean?
Graph data to display differences
Predictive Models
Provide measures of user performance without testing users
Useful when it is difficult to do user testing Estimate efficiency of different systems on
different tasks Most common models: GOMS family:
– GOMS Model, Keystroke Level Model
GOMS Model
Models cognitive processes used when interacting with systems
G oals: state the user wants to reach O perators: cognitive processes, physical
actions necessary to achieve goal M ethods: learned procedures for
accomplishing goals S election rules: used to choose between
available methods
Keystroke Level Model
Variation of GOMS model Provides numerical predictions Uses a standard set of times for main types
of operators used during a task Average time to perform certain tasks Compares times for task using different
startegies
GOMS Pros and Cons
Pros Allows comparative
analysis of different systems easily
Useful estimates to compare efficiency of different systems
Works well for short, well-defined tasks
Cons Only models tasks that
involve routine tasks Designed to predict
experts’ performance Difficult to predict for
average users Only predictions about
predictable behavior
Fitt’s Law
Useful for planning button locations, size and proximity
Predicts time to click on objects on a screen
T = k log2(D/S + .5), k ~ 100 msec– T: time to move hand to target– D: distance between hand and target– S: size of target
Bigger target = easier to reach
Summary
User Testing is the core of Usability Testing Controlled laboratory-like conditions Control independent variable(s) to predict
dependent variable(s) Experimental designs: different participants,
same participants, matched participants GOMS, Keystroke, Fitts’ can be used to
predict expert performance