Understanding Users Through Ethnography and Modeling

Jim JarrettSenior User Experience Architect and Manager

MedAssurant™

STC Technical Communication Summit 2010

3 May 2010

5:00-6:15pm

Reunion C

Understanding Users Through Ethnography and Modeling

Great product design starts with a deep understanding of the work that users do in the real world. Build your understanding through observation, interviews, surveys, and artifact collection. Share your understanding with KJ analysis, personas, and scenarios. Validate and prioritize your understanding with follow-up surveys.

Introduction and Conference Context 5 minutes

Ethnography and Data Collection 20 minutes

Modeling and Validation 30 minutes

Tips, Tricks, Tools, and Resources 10 minutes

Q&A 10 minutes

Who is Jim Jarrett?

Senior User Experience Architect and Manager

MedAssurant™

15 years at Rockwell Automationstarted as a multimedia developer in 1991other companies: Chiron, Roadway, IDD

Dozens of on-site observations in domains of medicalinformatics, industrial automation, and medical diagnostics.

Largest single user research project: 30 sites on 3 continents in 7 countriesacross 3 trips totaling 18 days on-site

Design for Six Sigma Blackbelt at Rockwell$2 million+ in hard savings over 2 years$24 million+ in revenue growth in same period

Multiple patents in UI for industrial automation

jim@jarrettinteractiondesign.com

www.JarrettInteractionDesign.com

LinkedIn.com/in/jarrettinteractiondesign

Twitter.com/JarrettUX

immersed in context

Ethnography and Modeling

Sketching

Expert Inspections

Usability Evaluation

Usability Institute

“Understanding Users Through Ethnography and Modeling”Monday 5:00-6:15pm Reunion C

“Discovering Usability Defects Through Expert Inspections” with Rich GuntherTuesday 8:00-9:15am Landmark B

“Improving Product Design Through Usability Evaluation” with Scott ButlerTuesday 9:45-11:00am Reunion F

“Sketching User Experiences with the Design Studio Method” with Brian SullivanWednesday 8:00-9:15am Reunion B

Inception

Elaboration

Construction

Transition

“… truth will sooner come out of error than from confusion.”Sir Francis Bacon

“Better wrong than vague!”

“An articulated guess beats an unspoken assumption.”Fred Brooks

Define Ethnography

ethno folk, people, nationgraphy writing, study of

“The scientific description of nations or races of men, with their customs, habits, and points of difference.”OED

“The study and systematic recording of human cultures.”Merriam-Webster

Qualitative, language-based, empirical, descriptive research.

Techniques include observation, interviews, surveys, logs and journals, gathering of artifacts and images.

Holistic and contextual.

“I prefer design by experts - by people who know what they are doing.” Don Norman

“… pay attention to what users do, not what they say… ” Jakob Nielsen

“A computer shall not harm your work or, through inaction, allow your work to come to harm.” Jeff Raskin

“What do users do and how do they talk about it?” Jared Spool

Market Research vs. User Research

Market Research

Markets and segments

Best ways to influence

Target demographics

Buying behavior

Message definition

Focus groups, surveys, and interviews

Competitive analysis

“Why do people buy?”

User Research

Individuals and context

Best ways to work

User profiles

Goals and motives

Task definition

Observation, surveys, and interviews

Domain analysis

“What do people do?”

Data Collection Process

1. Define the target of your research. What questions are you trying to answer? Who should you study?

2. Arrange visits. Get permissions and release forms. Schedule timing for your targets. Coordinate with sales and marketing.

3. Prepare your site teams.4. Two sites at a time, two observers at a time. Introduce, observe, interview, wrap-up, thanks. Collect artifacts and notice context. 90 minutes.

5. Process with project team. Immerse them in the experience and data.

6. Follow up. Thanks + additional questions.

7. Retarget.8. More visits, but not too many!

Example Observation 1

“I hate this [expletive deleted] history screen! I have to jump all around and read everything to find out the patient’s latest A1C measurement and see if she’s visited the pedorthist recently.”

Option 1

“How can we improve the history screen?”

Option 2

“I noticed you came to this screen a couple times while you were talking to the patient. What were you looking for? Can you walk me through your thinking?”

Don’t ask users what they want, watch what they do.

Users are not designers, and you are not a user.

Example Observation 2

During the discussion with the patient, the nurse refers to a small spiral bound note pad from time to time, sometimes making a quick notation. She also jots some short notes on a piece of scrap paper. While you are paying attention to those activities, you miss some of the conversation with the patient.

Option 1“Can you repeat what you said to the patient? I missed some of the details.”

Option 2

“While you were talking with the patient, you looked at your notebook and sometimes wrote something. Other times, you wrote on the scrap paper. What were you writing down? Why in one place versus the other?”

Synthesize a big picture with rich, textured detail.Invest in more research, not post-processing.

Example Observation 3

BUT!

Get permission. Don’t violate policies, regulations, or laws.

Get a signed release form if necessary.

Don’t let the camera get in the way of really observing the work.

bunch of reference material printed from the web

support, clinic, and social worker numbers

calendar with both personal and work events

brand name vs generic reference for common drugs

SOPs and comprehensive clinic and phsycian contacts

today’s call list

monitor turned to allow photo; avoid HIPAA issues

ID badge worn at all times

follow up notes from last call

A picture is worth a thousand memory slips.

Triggers rich memories from visit, immediately afterward when processing the data as well as long term.

Provides rich, visceral context for team members and stakeholders who weren’t there.

Particularly memorable ones can be used for personas.

Data Collection Challenges

Confidentiality, privacy, and trust.

HIPAA, intellectual property, security, legal, regulatory.

Complex environments.

International sites, hazardous work, longitudinal or rare work.

Redundant data.

Repetitive work, rare exceptions, awkward times.

Too much data.

Many sites, trip logistics, team availability.

Exhaustion.

This is exhilarating work, but can be grueling.

Define Modeling

“A systematic description of an object or phenomenon that shares important characteristics with the object or phenomenon.”American Heritage Science Dictionary

“A simplified or idealized description or conception of a particular system, situation, or process… that is put forward as a basis for theoretical or empirical understanding… a conceptual or mental representation of something.”OED

Concrete and detailed representations of observations.

Purposefully lossy: generalized but retaining meaningful details.

Domain models include KJ Analysis, Personas, Scenarios and a variety of others.

“models can frame the design problem… help designers understand the domain” Jared Spool

“The most effective behavioral models are distilled from interview and observation data of real users into an archetypal description of how a particular type of person behaves and what their goals are.” Kim Goodwin

“A mental model is a picture of how your end users are supported by what you are creating.”Indi Young

KJ Analysis

“Affinity on steroids.”

Outcomes:

Shared understanding

Relationships

Priorities

Process steps:

1. Define the question to be asked of the data.

2. Reduce the data set to 30 or fewer items through multiple rounds of simple voting.

3. Group related items.

4. Title the groups.

5. Group related groups.

6. Create headlines for the higher level groups.

7. Visually lay out the groups with all data items.

8. Show relationships between groups.

9. Vote for the most important title‐level groups.

10. Draw a conclusion from the diagram.

We don’t have a clear definition of

what a requirement is

No clear definition of what

the difference levels of

requirements are

There seems to be

a discrepancy on

how detailed the

rqmts need to be.

I can’t delineate

between BR’s and

MR’s.

It’s not clear

where a market

requirement stops

and an SR starts

I don’t understand what the customer

needs; we don’t understand what is

important to deliver

I like to be involved in the whole process

All roles should be

involved throughout the

entire process

VOCs are very valuable

Is seems like we are

spending a lot of time putting

the same data in multiple

places and coordinating to

ensure the data is in sync

The process shouldn’t impede the

progress of the project

We’re going to

need to do some

clarification of what is

design and what is a

requirements

We don’t have a

good definition of

what a

requirement is

It seems like different

RAs have a different view

of what a requirement

should be

With requirements at

a high level it is difficult

to come up with a

good estimate

Perhaps we should

have two different

levels of use cases

We don’t know what “good

enough” is

How do you define

success at the end

of each iteration

We have lost the

concept of critical

release

requirements

We aren’t validating what

we are building with the

customer

Need to get out of the

lab exercise of I know

what the answer is now

I need to go find a

customer to tell me they

have that problem

Need to know when

we come out the other

end, have we built the

feature that is necessary

for the customer.

You better think about

performance

requirements before you

start down the design

path

Need a better way

to bring customer

feedback in earlier in

the process

VOCs are very

valuable for

providing focus to

product issues

VOCS were very

valuable which gave

us a very good

process and means

to rank and prioritize

requirements

Get engineering and

test involved earlier on

before the rqmts are

finalized

The requirement

analyst needs to

own the rqmts all the

way through the

process.

It’s difficult to look at

requirements in

ReqPro so everyone

gives up

I like the ability to

add attributes (Req

Pro)

Requirements tools are

difficult to work with

A lot of things you test

for will not show up in

requirements for

instance failure modes

If you read only the

SRs it is difficult to

understand what the

hell it is supposed to

do.

Hundreds of hours have been spend on

requirements with very specific detail for

consistency. The gap we have is that how

are we going to do a consistent design

without that level of detail

If the requirements don’t

have the detail I need,

where do I get them?

When we took UI detail

out of the use case it

wasn’t clear what the

rqmt really meant

Where do we put this

extra UI information

It’s nice to have implementation details

as an example to understand what the

RA was thinking

UI Information is a good

way to communicate ideas

Anything so that people

aren’t waiting for other

people to get stuff done

We should not have

to have everything

written down and

signed off to hand it

over the wall

No good way ot

fimplementing and

managing features access

products

We need to find a

better way to illustrate

requirements on a

features basis

All requirements

need to be

coordinated across

products

The process shouldn’t

impede the progress of

the project

To be honest, it’s to

painful to do a change

I don’t know where to go to get the information I need

Theme: Gaining a better understanding of the

requirement processes and how customers should be

involved

Lessons Learned: Better defections is needed for the

levels of requirements and where to get requirement

details,

Anatomy of a KJ Diagram

Headline

Relationshipcause and effect or contradiction

Title

Raw statement

Title with most votesTitle with second most votes Title with third most votes

Question to ask?

Summary answer.

Time/datePlaceParticipants

Personas

“The aspect of a person's character that is displayed to or perceived by others.”OED

Tells a story

Concrete and specific

Memorable

Day-in-the-life narrative

Goals, tasks, and other behavior

Skills and demographics

“Personas are helpful in creating and iterating a design, building consensus, marketing the product, and even prioritizing bug fixes.” Kim Goodwin

Persona: Identity

Memorable name with role or user type

Memorable quote that captures the essence of persona’s goals and attitude

Memorable photo that strongly represents persona and role

Persona: Narrative

Detailed and specific

Concrete

Summarize a typical work day.

Persona: Skills

Identify skills that differentiate the different personas.

Rate the persona’s skill level for each.

Present visually and simply.

Persona: Details

Extend, generalize, and add specifics to narrative.

Enumerated and prioritized.

More analytical and processed than narrative.

Persona Quality

P Primary researchIs the persona based on contextual interviews with real customers?

E EmpathyDoes the persona evoke empathy by including a name, a photograph and a product-relevant narrative?

R RealisticDoes the persona appear realistic to people who deal with customers day-to-day?

S SingularIs each persona unique, having little in common with other personas?

O ObjectivesDoes the persona include product-relevant high-level goals and include a quotation stating the key goal?

N NumberIs the number of personas small enough for the design team to remember the name of each one, with one of the personas identified as primary?

A ApplicableCan the development team use the persona as a practical tool to make design decisions?

http://www.userfocus.co.uk/articles/personas.html

Scenarios

“An outline or model of an expected or supposed sequence of events.”American Heritage Dictionary

Envision the future.

Personas drive the action and point of view.

Begin with a triggering event, describes a sequence of actions, and the results of those actions.

Technology and solution-free.

A few key scenarios are better than a comprehensive set.

“A scenario supplies the context-of-use necessary to set the stage on the subtlety and nuance you'll need to get the mix just right.” Jared Spool

Scenario ExampleFirst step is the trigger that begins the scenario

Step by step, who does what, when, and why

Call out questions or observations each step brings to mind

Scenario vs. Use Case vs. User Story

A scenario is a concrete, sequential narrative about a specific workflow.

A use case is a generalization of a number of scenarios and can generate multiple scenarios.

A user story is a single statement that captures who, what, and why. Usually names a use case or scenario, or is a single step in one.

“An outline or model of an expected or supposed sequence of events.”American Heritage Dictionary

“… a collection of possible sequences of interactions between the system under discussion and its external actors, related to a particular goal.” Alistair Cockburn

“A description of desired functionality told from the perspective of the user or customer.” Mike Cohn

Other Models

Contextual Inquiry Flow Models Sequence Models Artifact Models Cultural Models Physical ModelsOrganizational ModelsProcess ModelsStakeholder ModelsConcept Models

Choose models that communicate the most important information to the most important stakeholders.

Keep the raw data around for many purposes: Explain or justify the models. Future data mining. Explorable body of knowledge.

Validate the Models

Validate KJ priorities with surveys. Rate and rank.

Much larger audience.

Validate personas with interviews. “Which do you identify with?”

“What parts don’t work?”

“Describe a day in your work life.”

Iterate throughout the lifecycle and across multiple projects.

Over-communicate the models and advocate their use.

surveymonkey

Yahoo! personas poster from article at UIE.com

Tools, Tips, and Tricks of the Trade

Get really good at taking notes while watching and talking. Don’t use a laptop.

Don’t bother recording. Do another visit instead.

Exception: Livescribe Pulse Smartpen

Take lots of pictures (if permitted).

“Revolving Door” technique. Leah Rader and Beth Toland

Card sorting. Information Architecture

Controlled Vocabularies

Journaling.

What To Do with All This Research

Share it!Accessible repository, posters, walkthroughs with stakeholders, conversations with SMEs, users, and customers.

Invent new products.How could current work be transformed or eliminated?

Frame all downstream work.Use cases, UI designs, and test scenarios based on the real world.

Validate the system.Does it support real people doing real work? Does it help them achieve their goals?

Simplify the system.Eliminate functions and workflows that don’t support the work.

Discover new avenues.What more research paths are opened up?

Resources

The Design of Design: Essays from a Computer ScientistFrederick P. Brooks, Jr.

Contextual Design: Defining Customer-Centered SystemsHugh Beyer, Karen Holtzblatt

Rapid Contextual Design: A How-to Guide to Key Techniques for User-Centered DesignKaren Holtzblatt, Jessamyn Burns Wendell, Shelley Wood

Commercializing Great Products with Design for Six Sigma, chapter 16 “KJ Analysis”Randy C. Perry, David W. Bacon

Designing for the Digital Age: How to Create Human-Centered Products and Services Kim Goodwin, Alan Cooper

The Persona Lifecycle: Keeping People in Mind Throughout Product Design John Pruitt, Tamara Adlin

AgileProductDesign.comJeff Patton

Writing Effective Use CasesAlistair Cockburn