qualitative evaluation material sources: dr. saul greenberg, univ. calgary john kelleher, it sligo
Post on 20-Dec-2015
220 views
TRANSCRIPT
2
Definition How to quickly evaluate prototypes by observing
people’s use of them How specific methods can help you discover
what a person is thinking about as they are using your system produces a description, usually in non-numeric terms may be subjective
3
Methods Methods
Introspection Extracting the conceptual model Observational Evaluation
simple observation think-aloud constructive interaction ethnography
Query via interviews and questionnaires Continuous evaluation via user feedback and field
studies
4
Designer tries the system (or prototype) out does the system “feel right”? most common evaluation method
Problems not reliable as completely subjective not valid as introspector is a non-typical user
Intuitions and introspection are often wrong
Introspection Method
5
Conceptual Model Extraction Show user the prototype / screen snapshots Have the user try to explain
what all elements are what they would do to perform a particular task
Excellent for extracting a novice’s understanding of system
Poor for examining system exploration and learning
6
Observational Evaluation
Quickly highlights difficulties
Verbal protocols valuable source of information
Can be used for rapid iterative development
Rich qualitative data
Observation can affect user activity and performance levels.
Analysis of data can be time-consuming and resource-consuming
Advantages Disadvantages
7
Direct Observation Evaluator observes and records users interacting with
design/system in lab
user asked to complete a set of pre-determined tasks a specially instrumented usability lab may be available
in field user goes through normal duties
Excellent at identifying gross interface problems Validity/reliability depends on how controlled /contrived the
situation is Three general approaches:
simple observation (See “User Testing”)
8
Thinking Aloud Protocol Asked the user to talk while doing the task
Have them tell what they are doing or trying to do as they perform the task
Have them read a set of instructions aloud before they begin the task Helps to get them going Can use the task description for this
Remind them to talk if they stop But be low-key about it
Spontaneous comments are best – don’t direct user comments with specific questions: “What do you think those prompts mean?” “Why did you do that?”
9
Pros and Cons ofThinking Aloud Pros
finds many usability problems finds why they occur (process data) small number of test users (3 to 5) usable early in development process requires little administrator expertise generates colourful quotes
Cons having to think aloud can influence user’s problem-solving
behaviour cannot provide performance data (bottom-line data) Cannot rely on user explanations (Maier, Nisbett)
10
Constructive Interaction Method
Two people work together on a task normal conversation between the two users is monitored
removes awkwardness of think-aloud
Variant: Co-discovery learning use semi-knowledgeable “coach” and naive subject together make naive subject use the interface
Results in naive subject asking questions semi-knowledgeable coach responding provides insights into thinking process of both beginner and
intermediate users
11
Recording Observations Evaluator may forget, miss, or misinterpret events Paper and pencil
primitive but cheap evaluators record events, interpretations, and extraneous observations hard to get detail (slow writing); coding schemes help…
Audio recording good for recording talk produced by thinking aloud/constructive
interaction Hard to synch user actions and annotations
Video recording can see and hear what a user is doing one camera for screen, another for subject (picture in picture) can be intrusive during initial period of use
13
Ethnography
Ethnographic study methods recognise that the investigator will have to interact directly with the subject, but while taking sufficient care to gain reasonably complete and objective information.
An ethnographic study will attempt to observe subjects in a range of contexts, over a substantial period of time making a full record using any possible means
(photography, video and sound recording as well as note-taking) of both activities and artefacts that the subject interacts with.
14
Survey Evaluation
Addresses users’ opinions and understanding of interface.
Can be made to be diagnostic Can be applied to users and
designers Questions can be tailored to
the individual Rating scales lead to
quantitative results Can be used on a large group
of users
User experience is important Low response rates (especially
by post) Possible interviewer bias Possible response bias Analysis can be complicated
and lengthy Interviews very time-
consuming
Advantages Disadvantages
15
Interviews Excellent for pursuing specific issues
vary questions to suit the context probe more deeply on interesting issues as they arise good for exploratory studies via open-ended questioning often leads to specific constructive suggestions
Problems: accounts are subjective time consuming evaluator can easily bias the interview prone to rationalization of events/thoughts by user
user’s reconstruction may be wrong Users recall away from context poor
16
Plan a set of central questions could be based on results of user observations gets things started focuses the interview ensures a base of consistency
Try not to ask leading questions Start with individual discussions to discover different
perspectives, and continue with group discussions the larger the group, the more the universality of
comments can be ascertained also encourages discussion between users
How to Interview?
17
Questions to elicit task detail
Questions Purpose
1. What do you do? Obtains the user’s goal
2. Why do you do it? Obtains method
3. How do you do it? Obtains the subtask. Used recursively for each subtask
4. What are the preconditions for doing this?
To find out what outside influences there are
5. What are the results of doing this?
To examine the product and see what the purpose is
6. What errors occur? Error capture
7. How do you correct them? Error correction
18
Challenges Users exhibit poor knowledge of exceptions (even
regularly occurring ones) Users exhibit poor recollection of work domain in
general Ethnography a possible solution for widening
social and organisation context Users exhibit little correlation between what they
said they would prefer and what they ultimately did prefer
Interviews not touch-stone for system acceptability
19
Post-observation interview to clarify events that occurred during system use perform an observational test create a video record of it have users view the video and comment on
what they did excellent for grounding a post-test interview
avoids erroneous reconstruction users often offer concrete suggestions
Retrospective Testing
20
Questionnaires & Surveys
Preparation “expensive”, but administration cheap can reach a wide subject group (e.g. mail)
Can be repeated to establish trends does not require presence of evaluator easy to quantify and analyse results only as good as the questions asked less flexible than interview Good for subjective responses
21
Establish the purpose of the questionnaire what information is sought? how would you analyze the results? what would you do with your analysis?
Do not ask questions whose answers you will not use! (e.g. how old are you?)
Determine the audience you want to reach Determine how you will deliver and collect the questionnaire
interview administered questionnaire on-line for computer users web site with forms surface mail (include SAE for better response) determine the demographics
Questionnaire Design
22
Questionnaire Design
Keep the number of questions low Only questions with answers that you can’t get other ways Only questions that will have a direct impact on functional
requirements Avoid asking for everything Avoid questions that users perceive as self-incriminating or
inferior Ask clear questions
precision in framing questions a problem Ask questions that users can answer validly and reliably
Does the user store information in this way? Does the user remember such information? Will the user be inclined to answer your question truthfully?
23
Styles of Questions
Open-ended questions asks for unprompted opinions good for general subjective information
but difficult to analyze rigorously
I found the following aspects particularly easy to use (please list 0–3 aspects):______________________________________________________________________________________________________
24
Style of Questions Closed questions
restricts the respondent’s responses by supplying alternative answers makes questionnaires a chore for respondent to fill in can be easily analyzed but watch out for hard to interpret responses!
alternative answers should be very specific Do you use computers at work:
often sometimes rarelyvs
In your typical day, do you use computers: over 4 hrs a day between 2 and 4 hrs daily between 1and 2 hrs daily less than 1 hr a day
25
Style of Questions Likert Scalar
ask user to judge a specific statement on a numeric scale (5, 6 or 7 point scale best)
scale usually corresponds with agreement or disagreement with a statement
The new drawing package is easy to use:
1 2 3 4 5Strongly agree agree neither agree nor disagree disagree strongly disagree
26
Style of QuestionsSemantic Differentials
sliding scale between opposing pairs of adjectives (5 or 7 point scale best).
Rate the Design and Go drawing package on the following criteria:
Very Quite Neutral
Quite Very
Simple 1 2 3 4 5 Complex
Professional 1 2 3 4 5 Unprofessional
Reliable 1 2 3 4 5 Unreliable
Attractive 1 2 3 4 5 Unattractive
27
Style of Questions Ranked
respondent places an ordering on items in a list useful to indicate a user’s preferences forced choice
Rank the usefulness of these methods of issuing a command(1 most useful, 2 next most useful..., 0 if not used)
__2__ command line__1__ menu selection__3__ control key accelerator
28
Style of Questions
Combining open-ended and closed questions gets specific response, but allows room for
user’s opinion
It is easy to recover from mistakes:
Disagree…………….Agree 1 2 3 4 5
comment: undo facility is really helpful
29
Pros and Cons of Questionnaires
Collect subjective user view of system Both quantitative and qualitative data Useful supplementary information Simple Cheap
Subjective data often unreliable
30
Continuous Evaluation Usually done in later stages of development
(i.e. beta releases, delivered system) Good for monitoring problems of system in actual use Problems can be fixed in next releasea) User feedback via gripe lines
users can provide feedback to designers while using the system Email and web forms special built-in gripe facility telephone hot line (help desks) suggestion box (in-house operations)
best combined with trouble-shooting facility users always get a response (solution?) to their gripes
31
Continuous Evaluationb) Case/field studies
careful study of “system usage” at the site E.g. Quicken by Intuit
good for seeing “real life” use external observer monitors behaviour or gets feedback via
methods described above Advantages and Dis-advantages
natural environment context retained (ecological validity) longitudinal studies possible cost distractions noise
32
Summary Observing a range of users use your system for specific tasks
reveals successes and problems Qualitative observational tests are quick and easy to do Several methods reveal what is in a person’s head as they are
doing the test Particular methods include:
Conceptual model extraction Direct observation
Simple observation Think-aloud Constructive interaction
Query via interviews, retrospective testing and questionnaires Continuous evaluation via user feedback and field studies