Ch. 12 Observing Users

Reece, Rogers, Sharp. Beyond human computer interaction.

Team 1:Andy, Nikhil, Vladimir, Sajay

12.1 Introduction

Learning Goals Benefits and challenges

of different methods How to observe How to deal with data Key issues in think-aloud,

diary studies, and interaction logging

Selecting and doing observations

Additional Content Critical Incident

Reporting: A type of diary observation

CIF: common industry format

Summative vs Formative Interaction Logging for the

Web Uzilla: A tool for web

usability testing by Andy

12.2 Observing in a controlled environment

i.e., usability testingPlan user and equipment location.Test the setup.Provide participants with a written

consent form and script of the study.Use think-aloud technique (p. 365) to find

out what participants are thinking.

Observing as an outsider

More objective than participant observation Lab equipment is in place Recording is continuous Analysis & observation almost simultaneous Care needed to avoid drowning in data Analysis can be coarse or fine grained Video clips can be powerful for telling story

Frameworks to guide observation

The Goetz and LeCompte (1984) framework:- Who is present? - What is their role? - What is happening? - When does the activity occur?- Where is it happening? - Why is it happening? - How is the activity organized?

The Robinson (1993) framework Space. What is the physical space like? Actors. Who is involved? Activities. What are they doing? Objects. What objects are present? Acts. What are individuals doing? Events. What kind of event is it? Goals. What do they to accomplish? Feelings. What is the mood of the group and of

individuals?

Frameworks to guide observation

Participant observation & ethnography

Must get co-operation of people observedInformants are usefulData analysis is continuousMove backwards and forwards from broad

to specific questionsCollect variety of dataHolistic approach – broad analysis

You need to consider

Goals & questionsWhich framework & techniquesHow to collect dataWhich equipment to useHow to gain acceptanceHow to handle sensitive issuesWhether and how to involve informantsHow to analyze the dataWhether to triangulate

12.4 Data collection techniques

Notes & still cameraAudio & still cameraVideoTracking users:

- diaries- interaction logging

Notes plus still camera Taking notes

Least technical way Most feasible way Could prove a challenge as one has to observe and

listen while taking notes. Writing skills such as writing speed, use of short

hand could prove useful One needs to translate and transcribe the written

content into useful information Other alternative are include another person

in the data collection process; use a laptop Use a camera to record images

Audio recording plus still camera

Audio recording Light and easy to carry Less intrusive than a video camera Transcribing long interview hours could be a

challenge Lack of a visual record Cannot be used in a noisy environment

Video Captures both visual and audio data Extremely intrusive Instruments could be expensive but

digicams/handheld camera are a cheaper option One may tend to focus only on what is seen

through the lens and ignore what has been missed out.

Sound may get muffled when recording under noisy conditions

Analysis of video data could be very time consuming.

Lets compare the Data-collection techniques – Table 12.2 page 376

12.5 Indirect Observation

Done when direct observation is not possible (obtrusive or evaluators cannot be present)

Tracking users Diaries Critical Incidents Interaction logging

Diaries

Classical paper based or using online forms linked to database etc.

Participant records what she did and experience she had while doing the task

Diaries collected from all the participants are then analyzed

Barry Brown’s research using diarys’ for HP design framework for devices used to capture or record information.

http://www.dcs.gla.ac.uk/~barry/papers/Diary%20study-chi%20paper.pdf

Advantages

InexpensiveSubjects could be tested location

independentExperimenter not required to be presentLarge amount of data could be gathered

Disadvantages

Relies on the reliability of the participantProblem of participant not remembering

actual experience of event (better or worse)

Subject is the experimenter and hence variability in data – not much control other than structures templates to collect information.

Interaction logging

Record all the interactions of the subject with the system (key presses, mouse movements, eye tracking, etc.) combined with audio and video logs.

Log is time stamped.Data can be analyzed to understand

cognitive aspects, user behaviors and reactions to variations in task etc.

Interaction logging - Uses

Useful for web studies.Testing Web layouts, displays, GUI

interfaces etc.Understanding the users’ reactions to

variations in tasks and layout combinations.

Testing the efficiency of the interface and its ease of use.

Interaction Logging on the Web

Server log file miss cached pages

To circumvent Web bugs – useful for

visitors at large Proxies – ala WebQuilt

http://guir.berkeley.edu/projects/webquilt/

Instrumented Browsers Andy’s Uzilla.net PARC’s WebLogger

Challenges Inferring user goals Analyzing paths

Triangulation Establishing the

magnitude of problems observed in testing from log files

12.6 Analyzing, Interpreting, and Presenting Data

12.6.1 Qualitative analysis to tell a story Hilight tapes, effective communication

12.6.2 Qualitative analysis for categorization Advanced linguistic analysis tools

12.6.3 Quantitative Data Analysis Measure learning, efficiency

12.6.4 Feeding the findings back into design Picture in a picture highlight tapes

Common Industry Format (CIF)

For summative testing (later stages in software development)

Subjective and Quantitative Measures On key tasks

Intended for vendors to communicate usability Boeing has adopted for all products and

suppliers

Uzilla: A product suite for web usability testing

Commercial venture by Andy Edmonds Overview talk SCIP 2002 Edmonds, K. A. (in press). Uzilla: A new tool for web usability testing. Behavior Research

Methods, Instrumentation and Computers. May, Special Issue of Soc. For Computers in

Psychology.

Automates test designInstruments data collectionAggregates resultsFree for student projects

Sarah Bly Interview

Unfinished book on non-speech audio in UI design

At Chi ’03 Visual Support for Conversations: People's Choice

Tutorial, Steve Harrison (Dyxsis D/R Consulting), Sara Bly (Sara Bly Consulting)To familiarize the participant with the underlying ideas and subtle design considerations that make shared drawing systems essential in many conversational settings.