1 evaluate—qualitative methods october 2, 2007 needs design implementevaluate
Post on 22-Dec-2015
217 views
TRANSCRIPT
11
Evaluate—Qualitative Evaluate—Qualitative MethodsMethods
October 2, 2007October 2, 2007
NEEDS
DESIGN
IMPLEMENTEVALUATE
22
EvaluationEvaluation
A little out of sequence due to A little out of sequence due to schedulingscheduling
Will get more implementation over Will get more implementation over next two weeksnext two weeks
Imagine you’ve implemented your Imagine you’ve implemented your applicationapplication
These are techniques you will need These are techniques you will need to design user study (end of to design user study (end of project)project)
33
Methods for evaluating Methods for evaluating system system QualitativeQualitative
– Rich, subjectiveRich, subjective– Exploratoring conceptsExploratoring concepts– More useful for earlier inputMore useful for earlier input
QuantitativeQuantitative– Precise, objective, repeatablePrecise, objective, repeatable– Demonstrating claimsDemonstrating claims– More useful at documenting More useful at documenting
improvementimprovement– Can be expensiveCan be expensive
44
For your projectFor your project
Will require aspects of both qualitative Will require aspects of both qualitative and quantitative methodsand quantitative methods– QualitativeQualitative
How users react to project, perceptions?How users react to project, perceptions?
– QuantitativeQuantitative How users perform on project?How users perform on project?
What would you improve on next What would you improve on next iteration?iteration?– Perhaps users’ perceptions of performance Perhaps users’ perceptions of performance
more important than actual valuesmore important than actual values– Elevator waiting storyElevator waiting story
55
DesignDesign evaluation evaluation methods!methods! Most important aspect of Most important aspect of
evaluation is upfront design!evaluation is upfront design!– Expensive to line up users, collect Expensive to line up users, collect
datadata– Design to collect right informationDesign to collect right information
Pick appropriate method for what Pick appropriate method for what you want to learnyou want to learn
66
Applying an evaluation Applying an evaluation methodmethod Determine the activity to observeDetermine the activity to observe Develop the methodDevelop the method Human subjects review approvalHuman subjects review approval Pilot test the methodPilot test the method Recruit participantsRecruit participants Collect the dataCollect the data Inspect & analyze the dataInspect & analyze the data Draw conclusions to resolve design problems, Draw conclusions to resolve design problems,
reflect on what you learnedreflect on what you learned Redesign and implement the revised interfaceRedesign and implement the revised interface
77
Demographic Demographic informationinformation Demographic dataDemographic data
– Age, gender, cultureAge, gender, culture– Task expertise, experienceTask expertise, experience– MotivationMotivation– Frequency of useFrequency of use– Education, literacy, trainingEducation, literacy, training
No matter what method, collect No matter what method, collect demographic datademographic data
88
Environmental Environmental informationinformation Besides info on the user, may also Besides info on the user, may also
need info on the operating need info on the operating environmentenvironment– Windows, Mac, Linux?Windows, Mac, Linux?– Firefox, Internet Explorer, Safari?Firefox, Internet Explorer, Safari?– Wired ethernet, wireless, modemWired ethernet, wireless, modem– Morning, afternoon, nightMorning, afternoon, night– Office, mobile, homeOffice, mobile, home
99
Qualitative methodsQualitative methods
““Discount” usability methodsDiscount” usability methods– Hueristic EvaluationHueristic Evaluation– Cognitive Walkthrough Cognitive Walkthrough
Questionnaire / SurveyQuestionnaire / Survey Think aloud protocolThink aloud protocol Co-discoveryCo-discovery Semi-structured interviewSemi-structured interview Deploy and observe in useDeploy and observe in use
1010
““Discount” usability Discount” usability methodsmethods Enable evaluation at early stage, Enable evaluation at early stage,
before prototype implementedbefore prototype implemented Conducted quickly, inexpensivelyConducted quickly, inexpensively Early evaluation investment saves Early evaluation investment saves
downstream development costsdownstream development costs– Hueristic evaluationHueristic evaluation– Cognitive walkthroughCognitive walkthrough
1111
Heuristic EvaluationHeuristic Evaluation
Fancy way to describe expert Fancy way to describe expert reviewreview– HCI expertHCI expert– Domain expertDomain expert
Expert review identifies usability Expert review identifies usability issues before implementationissues before implementation
Our grades on your homework are Our grades on your homework are form of heuristic evaluationform of heuristic evaluation
1212
Evaluation hueristicsEvaluation hueristics
Visibility of system statusVisibility of system statusMatch between system and the real worldMatch between system and the real worldUser control and freedomUser control and freedomConsistency and standardsConsistency and standardsError preventionError preventionRecognition rather than recallRecognition rather than recallFlexibility and efficiency of useFlexibility and efficiency of useAesthetic and minimalist designAesthetic and minimalist designHelp users recognize, diagnose, and recover Help users recognize, diagnose, and recover
from errorsfrom errorsHelp and documentation Help and documentation
1313
Heuristic evaluation Heuristic evaluation methodmethod Multiple experts individually review Multiple experts individually review
(around 5 experts get 75% (around 5 experts get 75% problems)problems)
Observer records issues, answers Observer records issues, answers questions, gives hintsquestions, gives hints
Conduct using low fidelity prototype Conduct using low fidelity prototype or task analysis with storyboards and or task analysis with storyboards and scenariosscenarios
Generate list of usability problems Generate list of usability problems according to hueristic compromisedaccording to hueristic compromised
1414
Hueristic Evaluation Hueristic Evaluation analysisanalysis After created list of problemsAfter created list of problems
– Rank severityRank severity– Estimate fixabilityEstimate fixability– Suggest possible fixesSuggest possible fixes
Analysis may involve larger teamAnalysis may involve larger team
1515
Hueristic Evaluation as Hueristic Evaluation as rigorous design reviewrigorous design review You can make a living out of You can make a living out of
doing Hueristic Evaluationdoing Hueristic Evaluation– Substantial consulting market for Substantial consulting market for
conducting Heuristic Evaluationconducting Heuristic Evaluation You may pay a consultant to do a You may pay a consultant to do a
Heuristic EvaluationHeuristic Evaluation– Know what you’re paying forKnow what you’re paying for– Especially the Severity, Fixability, Especially the Severity, Fixability,
Potential Fix aspectsPotential Fix aspects
1616
Learning more about Learning more about Hueristic EvaluationHueristic Evaluation You can learn to do a Hueristic You can learn to do a Hueristic
EvaluationEvaluation– http://www.useit.com/papers/http://www.useit.com/papers/
heuristic/heuristic/
1717
Cognitive WalkthroughCognitive Walkthrough
Have user imagine walking through Have user imagine walking through the process of using systemthe process of using system
Can use low-fidelity prototyping, Can use low-fidelity prototyping, partially implemented prototypepartially implemented prototype
Can use target user rather than Can use target user rather than expertexpert– Pluralistic walkthrough uses experts, Pluralistic walkthrough uses experts,
users, developersusers, developers Like a code walkthroughLike a code walkthroughC. Wharton et. al. "The cognitive walkthrough method: a practitioner's guide"
in J. Nielsen & R. Mack "Usability Inspection Methods" pp. 105-140.
1818
Walkthrough Walkthrough procedureprocedure Give user representation of interface Give user representation of interface
and taskand task– Can they discover how to accomplish goal Can they discover how to accomplish goal
with description of interface?with description of interface?– Can ask “From here, how would you like to Can ask “From here, how would you like to
accomplish…?”accomplish…?” Step through interfaceStep through interface
– User takes action, system provides responseUser takes action, system provides response– Describe actions not depicted in interface Describe actions not depicted in interface
representationrepresentation– Somewhat like Wizard of OzSomewhat like Wizard of Oz
1919
Stepping through Stepping through interfaceinterface Will user try to achieve the right goal? Will user try to achieve the right goal?
– Conceptual model of goals and tasksConceptual model of goals and tasks Will user notice correct action is Will user notice correct action is
available? available? – VisibilityVisibility– Understandability Understandability
Will user associate correct action with Will user associate correct action with the goal to be achieved? the goal to be achieved? – Aligning goals with sequence of actionsAligning goals with sequence of actions
If correct action performed, will user see If correct action performed, will user see progress toward solution? progress toward solution? – FeedbackFeedback
2020
Next assignmentNext assignment
Testing storyboard with one userTesting storyboard with one user– Effectively, this is a cognitive Effectively, this is a cognitive
walkthroughwalkthrough– Create storyboardCreate storyboard– Define taskDefine task– Step through with one userStep through with one user
2121
Questionnaires & Questionnaires & surveyssurveys User responses to specific questionsUser responses to specific questions Preparation is expensive, Preparation is expensive,
administration relatively cheapadministration relatively cheap Oral vs. writtenOral vs. written
– Oral provides interaction, followup, but Oral provides interaction, followup, but takes more timetakes more time
– Written more efficient, can provide Written more efficient, can provide quantitative dataquantitative data
2222
Designing questionsDesigning questions
Design questions with analysis in Design questions with analysis in mindmind– Closed format more precise, easier to Closed format more precise, easier to
analyzeanalyze Convert qualitativeConvert qualitativequantitative measuresquantitative measures You give categories to usersYou give categories to users
– Open-ended questions provide richer Open-ended questions provide richer feedback, longer to analyzefeedback, longer to analyze Users give you categoriesUsers give you categories
2323
Designing survey Designing survey questionsquestions Multiple choiceMultiple choice
– Collecting informationCollecting information Ordinal rankingOrdinal ranking
– Expressing relative preferencesExpressing relative preferences Likert scalesLikert scales
– Expressing personal reactionsExpressing personal reactions
2424
Closed format stylesClosed format styles
Orkut
MySpace
Rank frequency of usefrom5 – Most frequent1- Least frequent0 - Unused
___ facebook___ MySpace___ LinkedIn___ Orkut___ Other__________
Which social networkingsystems do you use?
Other_____________
Multiple choice Ordinal ranking
2525
Likert scalesLikert scales
Ask users to rate on a numeric scaleAsk users to rate on a numeric scale Odd number scale allows a neutral Odd number scale allows a neutral
midpoint (5- or 7-point scale)midpoint (5- or 7-point scale) Even number scale forces taking a Even number scale forces taking a
position (4- or 6-point scale)position (4- or 6-point scale) ““Anchors” give examples of points Anchors” give examples of points
along the scalealong the scale
2626
Example questionExample question
How important is the Berkeley-How important is the Berkeley-Stanford Big Game?Stanford Big Game?
Very ImportantNot Important
Most important event this Fall
Could notcare less
Maybe I’ll goif my friends go
2727
Closed FormatClosed Format
AdvantagesAdvantages– Clarify among Clarify among
alternativesalternatives– Easily Easily
quantifiablequantifiable– Eliminate useless Eliminate useless
answersanswers– Relatively quick Relatively quick
to administerto administer
DisadvantagesDisadvantages– Must cover whole Must cover whole
rangerange– All choices should All choices should
be similarly likelybe similarly likely– Don’t get Don’t get
interesting, interesting, “different” reactions“different” reactions
2828
Questions people can Questions people can answer about answer about themselvesthemselves What they doWhat they do How they do itHow they do it Opinions about current activitiesOpinions about current activities Complaints about current activitesComplaints about current activites Comparing one thing with anotherComparing one thing with another How often they have done How often they have done
something in the recent pastsomething in the recent past
2929
Questions people Questions people cannot answer about cannot answer about themselvesthemselves Predicting what they would do / Predicting what they would do /
like / wantlike / want Imagining a hypothetical scenarioImagining a hypothetical scenario Whether they would like a certain Whether they would like a certain
feature or productfeature or product Estimating how often they do Estimating how often they do
thingsthings
3131
Web-based survey Web-based survey toolstools SurveymonkeySurveymonkey
– http://www.surveymonkey.com/http://www.surveymonkey.com/ ZoomerangZoomerang
– http://info.zoomerang.com/http://info.zoomerang.com/ Allows free basic analysis, more Allows free basic analysis, more
advanced features for feeadvanced features for fee Can extend reach to large Can extend reach to large
number of respondentsnumber of respondents
3232
Thinking aloud Thinking aloud protocolprotocol Have subject “think out loud” Have subject “think out loud”
while performing taskwhile performing task Psychology to elicit cognitionPsychology to elicit cognition Requires training taskRequires training task Facilitator actively prompts if Facilitator actively prompts if
subject falls silent for more then subject falls silent for more then 10 secondss10 secondss– ““What are you thinking now?”What are you thinking now?”– ““So, you are trying to…?”So, you are trying to…?”– ““And now you are…?”And now you are…?”
3434
Co-discoveryCo-discovery
Have two people work on a task Have two people work on a task together (even though the task is together (even though the task is normally done by one person)normally done by one person)
Coordination with each other Coordination with each other naturally elicits cognitionnaturally elicits cognition
3535
Exercise: Two Exercise: Two volunteersvolunteers Never used Photoshop beforeNever used Photoshop before
3636
Think aloud and co-Think aloud and co-discoverydiscovery Valuable to evaluate tasks that Valuable to evaluate tasks that
require cognitionrequire cognition Time intensiveTime intensive Rich feedbackRich feedback Think aloud requires trainingThink aloud requires training
3737
Semi-structured Semi-structured interviewsinterviews Interactively asking questions (face-Interactively asking questions (face-
to-face, telephone)to-face, telephone) Give users chance to explain “why” Give users chance to explain “why”
to complement “what” they did, to complement “what” they did, subjective user’s viewpointsubjective user’s viewpoint
Can help with design questionsCan help with design questions– ““What improvements would you What improvements would you
suggest?”suggest?” Can be done individually or in groupsCan be done individually or in groups
3838
Semi-structured Semi-structured interviewsinterviews Begin with list of open-ended Begin with list of open-ended
questionsquestions– Ask all users these questionsAsk all users these questions– Let users elaborateLet users elaborate– Flexibility to ask follow-up questionsFlexibility to ask follow-up questions
Must audio-recordMust audio-record Interviewer should attend to user Interviewer should attend to user
(not notepad or laptop), use audio (not notepad or laptop), use audio record for data (note timestamps)record for data (note timestamps)
3939
Questionnaire IssuesQuestionnaire Issues
LanguageLanguage– Beware terminology, jargonBeware terminology, jargon
ClarityClarity– ““How effective was the system?” How effective was the system?”
(ambiguous)(ambiguous)
Avoid leading questionsAvoid leading questions– Phrase neutrally rather than positive Phrase neutrally rather than positive
or negativeor negative ““How easy or hard was it to accomplish How easy or hard was it to accomplish
the task?”the task?”
4040
Questionnaire Issues Questionnaire Issues (2)(2) Prestige bias Prestige bias
– People answer a certain way because they People answer a certain way because they want you to think that way about themwant you to think that way about them
Embarrassing questionsEmbarrassing questions– ““What did you have the most problems What did you have the most problems
with?”with?” Hypothetical questionsHypothetical questions ““Halo effect”Halo effect”
– When estimate of one feature affects When estimate of one feature affects estimate of another (e.g. intelligence/looks)estimate of another (e.g. intelligence/looks)
– Aesthetics & usability, one example in HCIAesthetics & usability, one example in HCI
4141
InterviewsInterviews
DisadvantagesDisadvantages– Subjective viewSubjective view– Interviewer(s) can bias the interviewInterviewer(s) can bias the interview– Problem of inter-rater or inter-Problem of inter-rater or inter-
experimenter experimenter reliabilityreliability (agreement)(agreement)
– Time-consumingTime-consuming– Hard to quantifyHard to quantify
4242
Pilot test observation Pilot test observation methodmethod Pilot test method with some target Pilot test method with some target
usersusers– Debug the questions, methodsDebug the questions, methods– Also debug logisticsAlso debug logistics– Don’t count pilot data in analysisDon’t count pilot data in analysis
Make changes now before Make changes now before collecting data (want method for collecting data (want method for collecting data to be consistent)collecting data to be consistent)
4343
Methods used in Methods used in combinationcombination Mix of closed format, open-ended Mix of closed format, open-ended
questionsquestions Surveys, questionnaires often Surveys, questionnaires often
used with quantitative used with quantitative performance measures to assess performance measures to assess how users feel about interactionshow users feel about interactions
4444
Mechanics of user Mechanics of user testingtesting Readings give more detailed nuts Readings give more detailed nuts
and boltsand bolts Common sense structuring of the Common sense structuring of the
experience to help it run experience to help it run smoothlysmoothly
4545
Analyzing qualitative Analyzing qualitative datadata Rich, open-ended dataRich, open-ended data Goal: Structure to characterize, Goal: Structure to characterize,
describe, summarize datadescribe, summarize data Sounds harder than it isSounds harder than it is
4646
Analyzing qualitative Analyzing qualitative datadata Exercise to immerse in dataExercise to immerse in data
– Develop categories to countDevelop categories to count RangeRange AverageAverage
– Identify common patternsIdentify common patterns Allows identifying the interesting, Allows identifying the interesting,
unusual, exceptionsunusual, exceptions Also look for correlationsAlso look for correlations
4747
Exercise: Analyzing Exercise: Analyzing conceptual map of conceptual map of BerkeleyBerkeley Example of rich, qualitative dataExample of rich, qualitative data See if we can detect some See if we can detect some
patternspatterns Characterize set of qualitative Characterize set of qualitative
datadata
4848
Berkeley mapBerkeley map
Number of features?Number of features? Format of mapFormat of map Common featuresCommon features
– LandmarksLandmarks– RoadwaysRoadways
Unusual featuresUnusual features AssessmentsAssessments CorrelationsCorrelations
4949
Qualitative analysisQualitative analysis
Start with things you can countStart with things you can count– Average, range, medianAverage, range, median
Look for patterns that are in Look for patterns that are in commoncommon
Recognize features that are Recognize features that are unusual, interestingunusual, interesting
Look for correlationsLook for correlations Reflect on what the data is sayingReflect on what the data is saying
5050
Qualitative study of Qualitative study of your projectyour project What do you want to learnWhat do you want to learn
– User reactions, perceptionsUser reactions, perceptions– Conceptual model problemsConceptual model problems– Areas to improve designAreas to improve design– Does the design work?Does the design work?
5151
Next timeNext time
Quantitative methodsQuantitative methods ReadingsReadings
– ""A face(book) in the crowd: social A face(book) in the crowd: social Searching vs. social browsing" Searching vs. social browsing"
– ""iPod distraction: effects of iPod distraction: effects of portable music-player use on portable music-player use on driver performance"driver performance"
Questions on Project Proposal Questions on Project Proposal assignment?assignment?