requirements gathering & task analysis – part 2 of 5

Requirements Gathering & Task Analysis – Part 2 of 5

Why, What and How – Methods

This material has been developed by Georgia Tech HCI faculty, and continues to evolve. Contributors include Gregory Abowd, Jim Foley, Diane Gromala, Elizabeth Mynatt, Jeff Pierce, Colin Potts, Chris Shaw, John Stasko, and Bruce Walker. Comments directed to [email protected] are encouraged. Permission is granted to use with acknowledgement for non-profit purposes. Last revision: Jan 2014.

mailto:[email protected]

UI Design - Georgia Tech

How – Some (Not All) User & Task Analysis Methods

ObservationInterviewsEthnographySurveys/QuestionnairesFocus groupsStudy documentationStudy competitive productsData recording - automatedData recording – diary (log) study


Formative & Summative Evaluation

• Formative evaluation Conducting this process to help guide the formation (ie, design) of a UI

• Summative Evaluation Conducting this process to help summarize

(sum up) the effectiveness of an existing or developmental UI

• Many of the user & task analysis techniques can be used for both formative and summative evaluation Our focus right now is on formative evaluation Will revisit some of the methods again later


Observation - From a Distance

• Watch users do what they do Step-by-step procedures Pauses to figure out what to do Mistakes/errors

• Do this today or tomorrow Automatic Checkout, ATM, parking

meter Don’t scare off the observees!!

• Record (with permission)


Observation – up Close

• Often combined with “Thinking Out Loud”

• Encourage user to verbalize what they are thinking Put them at ease first! Not every user is good at this Hard for user to do for long time; need

breaks• Record (with permission)• Up close and at a distance not mutually

exclusive


Pros and Cons?

• Pros

• Cons


Interviews

• Structured – “Just the facts” Fixed set of questions

– How do you know what questions to ask?

May miss subtleties and big picture

• Unstructured – A conversation Go with the flow

– May learn what questions to ask• Semi-structured – start with

focused questions, move to open-ended discussion Good balance, often appropriate Training – process + domain

knowledge• Do some homework first


Interviews• Know why you are asking questions - don’t

waste time• Plan for effective question types

– How do you perform task x?– Why do you perform task x?– Under what conditions do you perform task x?– What do you do before you perform…?– What information do you need to…?– Whom do you need to communicate with …?– What do you use to…?– What happens after you…?– What is the result or consequence of…?– What is the result or consequence of NOT…?


Interviews: Typical Open-Ended Questions

• Why do you do this (whatever the task is you are studying)

• How do you do this? Gets at task-subtask structure Then ask about each subtask

• Why do it this way rather than some other way? Attempts to get user to explain method so you

can assess importance of the particular way of doing task

• What has to be done before you can do this? To understand sequencing requirements


Interviews: More Typical Open-Ended Questions

• Please show me the results of doing this• Do errors ever occur when doing this?

If answer is ‘yes,’ then learn why occur How do you discover the errors, and how do

you correct them?• Encourage digressions; ask for

elaborations Pause before asking next question

• What else should I have asked you?


Interviews: Domain Experts

• Expert may describe how it should be done Not necessarily how it is done :-)


Interviews: Users vs. Management

• Folks who are not in the trenches doing the work may have a different view than those who are actually doing whatever it is you are studying.

• In organizations, can be political


Ethnography

• Similar to Observation, but more intense• Observe people in their cultural context

Because behavior most meaningful in context• Immerse oneself in situation you want to

learn about (has anthropological and sociological roots); use In person observation Interviews – formal and over coffee Audio/video recording – after trust develops Learn to do the work yourself! (biggest difference

from observation)


Ethnography: Types of Findings

• Can be both Qualitative

– Observe trends, habits, patterns, …

Quantitative– How often was something done, what per

cent of the time did something occur, how many different …


Ethnography: Drawbacks

• Time required Can take weeks or months for large systems

• Scale Most use small numbers of participants just to

keep somewhat manageable• Type of results

Highly qualitative, may be difficult to present/use

• Acquired skill – “learn by doing” Identifying and extracting “interesting” things

is challenging


Questionnaires

• Make questions clear and specific• Ask some closed questions with

range of answers Sometimes also have a no opinion

option, or other answer option• Do test run with a few people• Use computer-based tools


Questionnaires – Likert Scale

• Seven-point Likert Scale (use odd #)

• Slightly leading question Could be seen as favoring one answer

– Help was worthwhile– Agent intrusive

• To avoid issue….


Questionnaires – Semantic Differential

Annoying

Easy to use

Attractive

Secure

Helpful

Modern

Pleasing

Difficult to useUnattractive

Not secureUnhelpful

Dated

X

X

X

X


Questionnaires – Typical Questions

• Rank the importance of each of these tasks (give a list of tasks)

• List the four most important tasks that you perform (this is an open question)

• List the pieces of information you need to have before making a decision about X, in order of importance

• Are there any other points you would like to make? (open-ended opinion question; good way to end)

• Same questions can be used in interview and in questionnaire; difference is in follow-up opportunity

Questionnaire Pros/Cons

Pros Cons



Focus Groups

• Group of individuals - 3 to 10 Use several different groups with different roles or

perspectives Careful about power relationships – separate groups Careful about few people dominating discussion

• Use structured set of questions More specific at beginning, more open as progresses Allow digressions before coming back on track

• Relatively low cost, quick way to learn a lot• Audio or video record, with permission• Blog – open-ended focus group• Pros/cons?


Study Documentation

• Similar in some ways to the expert interview

• Often describe how things should be done rather than how they are done Try to understand why not done “by the

book”


Study Competitive Products

• Looking for both good and bad ideas Functionality UI style

• Try to understand why are successful or unsuccessful

• Measure task completion times for new and experienced users Upper bounds for new product


Data Recording – Time-stamped Event Log

• High-level, such as Web page visits Commands used

• Low-level, such as Keystroke and mouse events Device/person location; think Aware

Home• Lots of data - need analysis tools• Pros/Cons?


Data Recording - Manual (Log Study)

• Participants keep a log for days or weeks

• Asked to record relevant events• May do follow-up interview• Lot of detailed text to analyze after

the fact

• Pros/cons


Methods to Use Depends on…

• Resources• Current knowledge of tasks and users• Context

Can’t use talking out loud if tasks involve two people working together

• Essential to use some methods• Not likely you will use all methods• Which best for class project?• Understand pros/cons of each

On own and relative to other methods

End of Part 2



Ethnography: Observations

• Carefully observe everything about users and their environment Structure and language used in work Individual and group actions Culture affecting work Explicit and implicit aspects of work; what is

said, what is left unsaid• Example: Office work environment

Business practices, rooms, artifacts, work standards, relationships between workers, managers, …


Why is Ethnography Useful?

• Helps gain rich assessment of user needs Help to define requirements

• Uncovers true nature of user’s job Discovers things that are outside of job

description or documentation• Allows you to play role of end-user better

Can sit in when real users not available• Open-ended and unbiased nature

promotes discovery

requirements gathering & task analysis – part 2 of 5

Documents

task x

user task analysis techniques

loudencourage user

tasksubtask structurethen

focused questions

particular way

domain expertsexpert

summative evaluationour