evaluating the usability of web-based applications a case study of a field study sam j. racine, phd...
DESCRIPTION
How do we measure usability? Goals determine techniques Techniques determine test designs Test designs determine results Results must be interpretedTRANSCRIPT
![Page 1: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/1.jpg)
Evaluating the Usability of Web-based ApplicationsA Case Study of a Field Study
Sam J. Racine, PhDUnisys Corporation
![Page 2: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/2.jpg)
What does “Usability” mean?The measure of how a given user operates a given interface in a given context effectively, efficiently, and with satisfaction
![Page 3: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/3.jpg)
How do we measure usability?Goals determine techniques Techniques determine test designsTest designs determine resultsResults must be interpreted
![Page 4: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/4.jpg)
What is “Contextual Inquiry”?Ethnographic observation of users in their environmentFlexible methodology that produces open-ended resultsTechnique that is excellent for beginning or revisiting a UI design process
![Page 5: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/5.jpg)
What does contextual inquiry provide?An understanding of what users are really doing day to day, in order to translate their tasks into an effective UI design
![Page 6: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/6.jpg)
Contextual inquiry applied
Unisys LMS Enterprise Services (ES) a web-based cargo management application
• Basis for “sister” applications
• Basis for other web-based applications in similar environments and markets
• Desire to build on our successes (and failures)
![Page 7: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/7.jpg)
Our goalsDetermine the real-world value that users (not customers) assign to our applicationLearn what users really do, not just what they say they doSort through the “lore” from the “reality”
![Page 8: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/8.jpg)
Our technique
Field study methodology to learn about• users: their background, needs, and working environment
• interactions with the application: types of tasks, frequency, and completion time
• available support material and derived work-arounds
• ‘training’ and to whom users go for help
![Page 9: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/9.jpg)
External factors affecting test designCustomer site
• Management
• Employees
• Contractors
• Schedules
StakeholdersLogistics
![Page 10: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/10.jpg)
Internal factors affecting test designExpertisePersonnel availabilityBudgetMore stakeholdersMore logistics
![Page 11: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/11.jpg)
Our test designTwo weeks, two evaluatorsFirst week observation; second week analysisFirst week ‘note taking;’ second week discussion and response to users’ questionsOpen-ended details:
• Daily revamp of test design
• Intervening weekend finalized second week’s details
![Page 12: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/12.jpg)
“Rules” for data gatheringNote comments verbatimFor a survey, repeat questions exactlyFor information, customize approachNote response and participant and contextRespect each evaluator’s approachNote what users do as well as what they sayBe flexible
![Page 13: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/13.jpg)
“Rules” for sorting data
Pay attention to what users do• but don’t discount what they say
Allow categories to emerge from dataCategorize after collecting data
• no preconceptions
Assign weight according to user needs and yours separatelyDon’t dismiss anomalies
• measure against the participant and context
![Page 14: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/14.jpg)
“Rules” for dealing with users and stakeholders
Work according to their comfort level• not yours
Treat each user as a CEOKnow communication valued by stakeholdersBe prepared with multiple arsenals of communication
• nonverbal like diagramming or picture drawing
• bring a camera
Appreciate, don’t bribe
![Page 15: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/15.jpg)
The test
Step 1: PreparationEstablish climate and expectations
• Announcement to participations• Confirmation of arrival logistics• Practice observation
Step 2: Observation onlyGather data
• “Take a note”• No “answers” or “corrections”• Participation survey and identification• Investigation of expressed concerns by managers and users• Note everything!
![Page 16: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/16.jpg)
The test
Step 3: AnalysisSort observations according to organic categories
• Sites of difficulty• Clustered impressions• Patterns• Preliminary Findings• Training opportunities
Step 4: Directive ActivitiesTest categories and verify findings
• Cognitive walkthroughs• Directed activities• Demonstrations
![Page 17: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/17.jpg)
Findings: Data
Internet is a new creature• changed expectations
• changed operation
Training and orientation two different thingsUsers value accuracyDomain experts require supportive UI“Old” eyes are everywhere
![Page 18: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/18.jpg)
Findings: MethodologyA ‘true’ source exists for most complaintsMultiple sources doesn’t guarantee validityOrder in which data is discovered affects interpretationSignificance is assigned by users and evaluators
![Page 19: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/19.jpg)
What did we do well?Had patience, patience, patienceStaggered across times and shiftsSwallowed our pride and rejected seductionAsked for helpMade the most of what we hadTimed our exitBrought a camera
![Page 20: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/20.jpg)
What could we have improved?Extended to multiple sitesSought a different quarterBetter pre-testing, identification, and administration of surveysBrought a video camera
![Page 21: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/21.jpg)
RecommendationsContextual inquiry requires
• observation
• communication
• detection
• theoretical foundation
In other words, use a professional!
![Page 22: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/22.jpg)
Questions?
![Page 23: Evaluating the Usability of Web-based Applications A Case Study of a Field Study Sam J. Racine, PhD Unisys Corporation](https://reader034.vdocument.in/reader034/viewer/2022051307/5a4d1b947f8b9ab0599c2d8d/html5/thumbnails/23.jpg)
Thank you!Sam Racine, Unisys [email protected]