evaluation methods april 20, 2005 tara matthews cs 160
Post on 21-Dec-2015
216 views
TRANSCRIPT
![Page 1: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/1.jpg)
Evaluation Methods
April 20, 2005
Tara Matthews
CS 160
![Page 2: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/2.jpg)
In 160 We’ve Covered…
• Task Analysis & Contextual Inquiry
• Cognitive Walkthrough
• Heuristic Evaluation
• WOZ usability study w/ paper prototypes
![Page 3: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/3.jpg)
There are many more methods…
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study
• Automated observation user study
• Experimental simulation
• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 4: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/4.jpg)
How to chose a method?
• Stage of study– formative, iterative, summative
• Pros & cons
• Metrics– depends on what you want to measure
• Qualitative vs. quantitative
• Research perspective– CS vs. psychology vs. sociology
![Page 5: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/5.jpg)
Pros & Cons
• Realism
• Precision
• Generalizability
• Time & cost
• Researcher expertise
![Page 6: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/6.jpg)
Methods
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study
• Automated observation user study
• Experimental simulation
• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 7: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/7.jpg)
Survey
• Online / paper questionnaires distributed to target audience
• Can be used to– tabulate quantitative data– gather qualitative feedback (opinions,
feelings, etc.)
• Useful at any time in study
![Page 8: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/8.jpg)
Survey
• Pros– Easy to get a large number of responses.– Quick and easy to conduct.– Highly generalizable.
• Cons– Self-selection.– Participants often only offer enough information to
answer the question.– Can miss details.– Low in realism and precision.
![Page 9: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/9.jpg)
Interview
• Evaluators formulate questions on the issues of interest.
• Interview representative users, asking them these questions in order to gather information desired.
• Interviewer reads questions to user, who replies verbally; interviewer records responses.
![Page 10: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/10.jpg)
Interview
• Pros– Quick and easy to conduct.– Gives designer quick feedback on a range of ideas.– Can get a person’s initial reaction to an idea.– Can get detailed information from a person.
• Cons– Often takes place away from natural setting.– Question wording or interviewer “body language” can bias
answers.– High probability of false positives and missed problems (e.g.,
users may not have a clear idea of how an app will be used).– Can miss details if interviewer doesn’t know what issues to
draw out.
![Page 11: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/11.jpg)
Controlled Lab Experiment
• In lab, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system.
• Example:– in lab, show users 4 versions of a website:
• blue, yellow, red, and black text
– measure time to find specific words– compare
![Page 12: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/12.jpg)
Controlled Lab Experiment
• Pros– Provides precise, quantifiable data.– Easier to draw inferences from data.– Relatively quick.– Can get a medium-sized number of participants.
• Cons– Short duration of a lab experiment may not be enough
to allow users to become accustomed to an app.– Not a natural setting – interaction may not be normal.
![Page 13: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/13.jpg)
In-lab Observation
• Participants come to lab to "use" an interface• Given sample tasks to complete with it• Evaluators observe and possibly audio- or
videotape• Participants may "think out loud"• Can use lo-fi prototype (for a project in the
design stage) to an almost-complete interface• Evaluators note participants’
– emotions, exclamations, facial expressions, and other "qualitative" data
– take note of quantitative data such as time to complete a task or number of errors
![Page 14: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/14.jpg)
In-lab Observation
• Pros– Relatively quick.– Can get a medium-sized number of participants.
• Cons– Observations are subjective and error prone.– Short duration of lab observation is not enough time
for user to get accustomed to using the interface.– Not a natural setting – interaction may not be normal.
![Page 15: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/15.jpg)
Controlled Field Experiment
• In natural setting, manipulate one feature of a system to assess the causal effects of the difference in that manipulated feature on other behaviors of the system.
• Example:– Participants use 3 different input devices in
their own office: mouse with 1, 2, or 3 buttons– Perform a set of tasks– Measure differences
![Page 16: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/16.jpg)
Controlled Field Experiment
• Pros– Less intrusive than most other evaluation methods.– Provides more precise data than field observation.– Can observe natural behavior of user (though some
part of the system will be controlled/unnatural).
• Cons– More intrusive than field observation.– Less natural than field observation.
![Page 17: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/17.jpg)
Field Observation Study
• Evaluator makes direct observations of “natural” systems
• Takes care to not intrude on / disturb those systems
• A.K.A. “ethnography”
![Page 18: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/18.jpg)
Field Observation Study
• Pros– Only way to observe natural behavior of user &
interaction between user & tools.
• Cons– Difficult and time consuming.– Hard to get permission to observe people.– Observations are subjective and error prone.– Cannot make strong interpretations from
observations.– Not very generalizable.
![Page 19: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/19.jpg)
Heuristic Evaluation
• Pros– Quick and easy.
• Cons– Nielson’s heuristics may not be as relevant to
non-GUIs.– Results in false positives in missed problems,
especially when experts are not part of target audience.
![Page 20: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/20.jpg)
Cognitive Walkthrough
• Pros– Quick and easy.
• Cons– Results in false positives and missed
problems when evaluator is different from target audience.
![Page 21: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/21.jpg)
Automate Observation Study
• Techniques include– video or audio recording of user– pop-up screens– screen shots– time logging– log users actions (collecting statistics about
detailed system use)
![Page 22: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/22.jpg)
Automate Observation Study
• Pros– Eases burden on observers for data collection
& analysis.
• Cons– Setup is often more time-consuming to
complete.– Harder to get approved if it involves analysis
of videotape or audiotape.– May miss nuanced/interpretive details.
![Page 23: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/23.jpg)
Experimental Simulation
• In-lab experiment that is as much like some real situation as possible.
• Example:– ground-based flight simulator– behaves as closely as possible to a real flight– still under researcher control
![Page 24: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/24.jpg)
Experimental Simulation
• Pros– Still fairly precise.– More realistic than in-lab experiment.
• Cons (same as lab exp.)– Short duration of a lab experiment may not be
enough to allow users to become accustomed to an app.
– Not a natural setting – interaction may not be normal.
![Page 25: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/25.jpg)
Claims Analysis
• Claim = statement that a certain aspect (button, scrollbar) of a design has psychological implications reflected in how capable a user is in using that design
• UI artifacts are listed along with their design features & pros/cons
• Helps– select among alternative designs– clarify questions to be analyzed through user testing
by stating how the design should work (in claims)
![Page 26: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/26.jpg)
GOMS
• A method to describe user tasks and how a user performs those tasks with a specific interface design
• Views humans as information processors– Small number of cognitive, perceptual, and motor operators
characterize user behavior• To apply GOMS:
– Analyze task to identify user goals (hierarchical)– Identify operators to achieve goals– Sum operator times to predict performance
• GOMS = – Goals: What a user wants to accomplish– Operators: Cognitive or physical actions that change the state of
the user or the system– Methods: Groups of goals and operators– Selection rules: Determine which method to apply
![Page 27: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/27.jpg)
GOMS
• Pros– Predict human performance before committing to a
specific design in code or running user studies– Many studies have validated the model (it works)
• Cons– Assumes error-free, skilled user behavior– No formal recipe for how to perform analysis– Significant time investment
![Page 28: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/28.jpg)
Computer Simulation
• Creating a complete & closed system that models the operation of the concrete system without users.
• Example:– geophysical process going on in connection
with the eruption of Mount St. Helens
![Page 29: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/29.jpg)
Computer Simulation
• Pros– Supposedly high in realism (depends on
accuracy of data/system replication)
• Cons– Low in precision & generalizability
![Page 30: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/30.jpg)
Formal Theory
• Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest.
• Pros– Relatively generalizable
• Cons– Not realistic or precise
![Page 31: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/31.jpg)
How to chose a method?
• Stage of study• Pros & cons
– Realism– Precision– Generalizability– Time & cost
• Researcher expertise• Metrics• Qualitative vs. quantitative• Research perspective
![Page 32: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/32.jpg)
Methods
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 33: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/33.jpg)
Early Stage
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 34: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/34.jpg)
Early Stage
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 35: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/35.jpg)
Iterative & Summative Stages
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 36: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/36.jpg)
Iterative & Summative Stages
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 37: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/37.jpg)
Realism
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 38: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/38.jpg)
Realism
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 39: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/39.jpg)
Precision
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 40: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/40.jpg)
Precision
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 41: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/41.jpg)
Generalizability
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 42: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/42.jpg)
Generalizability
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 43: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/43.jpg)
Time & Cost
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 44: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/44.jpg)
Time & Cost
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 45: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/45.jpg)
Researcher Perspective
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 46: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/46.jpg)
Metrics: examples
• Traditional GUIs:– efficiency (time to complete task)– accuracy (# of errors)– simplicity
• Peripheral Displays:– awareness (recall)– distraction (dual-task behavior)– aesthetics
![Page 47: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/47.jpg)
Peripheral Displays
• Survey• Interview• Controlled-lab
experiment• In-lab observation• Controlled field
experiment• Field observation
study• Heuristic Evaluation
• Cognitive Walkthrough• Contextual Inquiry• Automated
observation user study• Experimental
simulation• Claims analysis• GOMS• Computer simulation• Formal theory
![Page 48: Evaluation Methods April 20, 2005 Tara Matthews CS 160](https://reader031.vdocument.in/reader031/viewer/2022013004/56649d605503460f94a40d83/html5/thumbnails/48.jpg)
Questions?