Gathering User Data

IS 588

Dr. Dania Bilal

Spring 2008

Getting to know users

• Setting goals for data collection

• Working with research participants– Consent forms (IRBs) (signed by adults)– Assent forms (signed by children)

• Choosing from various methods– Quantitative and qualitative

Methods

• Interviews

• Focus groups

• Observation

• Surveys and survey questionnaires

• Field studies (in Evaluation)

• Transaction logs

• Experiments (in Evaluation)

Interviews

• Structured interviews– Predetermined closed questions established

• Unstructured interviews– Open questions

• Semi-structured interviews– Open and closed questions

Planning an interview

• Develop a set of questions based on goals and objectives of data gathering

• Open vs. closed questions

• Combination of open and closed questions

• Questions to be written according to user’s level of understanding– Explanations provided as needed

Types of interviews

• Face-to-face individual interviews with answers handwritten on interview question form

• Face-to-face and audio-recorded interviews

• Telephone interview with or without audio-recording

• Diaries as basis for interviews• Qualitative method

Focus groups interview

• Qualitative method• Focus group interview

– 3-10 people– A group (or different groups) with a focus– May be from diverse user groups representative of target

population– Requires a well-trained facilitator– Gather opinions, not behavior– Based on preset questions and agenda

• Facilitator can follow up on unanticipated issues, ideas, etc.– Interviews are recorded, transcribed

• Participants can be invited to listen to recording or read transcribed interview to explain comments or clarify certain responses

– Can be expensive and time consuming

Interview questions

• Exploratory: Open questions– Typically unstructured questions

• Possible answers: closed questions– Typically structured questions

• Combination of both types

Guidelines for interview questions

• Ask clear, well-worded, non-leading questions

• Do not ask double barrel questions

• Ask questions that encourage participants to state opinion, express feelings, as applicable

• Pilot-test, refine, and use questions

Running the interview

• Introduce yourself• Introduce purpose of interview

– Participant to read and sign consent form– Mention time interview may take in consent form and verbally– Participant’s signature confirms willingness to participate

• Ask easy questions first such as demographic ones (warm-up session)– Opening questions

• Main questions• Closed questions• Thank you note

Observation

• Quantitative technique

• Gather information about behavior

• Provide information about users’ context, tasks, goals, success, difficulties, etc.

• Two types: – Direct– Indirect

Direct observation

• In a user’s field (natural setting)• In a controlled environment (setting chosen by

observer: lab, café, etc. )• In the field: Provides detail that can’t be captured

through qualitative method• Requires good planning: structure and focus

– Think of who you are observing– What you are observing– Where the observation is taking place (get to know

the environment)– See text, p. 325 for detailed framework.

Direct observation: Controlled environment

• Typically takes place in a usability lab.

• Can be performed in a setting using a portable usability lab and/or software to capture user actions, verbalization, and feelings

• Detail will be covered in Evaluation

Indirect observation

• Tracking users’ activities– Transaction logs– Diaries/journals

• Advantages/disadvantages of interaction logs

• Advantages/disadvantages of diaries/journals

Choosing and combining methods

• Variables to consider– Goal and focus of data gathering– Participants involved (who they are, how

many, where they are, accessibility)– Time for data collection, transcription, coding,

and analysis– Resources available– Other?

Survey and questionnaires

• Technique for collecting data and users’ opinions

• Used to capture data from group spread across a wide or small geographic area or place

• Level of success based on type and nature of questions asked

Questionnaires

• Questions can be like those used in structured or semi- structured interviews

• Closed questions

• Open-ended questions

• Combination of both types of questions

Questionnaire design

• Introduction• Instructions on how to complete

questionnaire• Question sequencing• Length of questionnaire• Check boxes• Scales (Likert; semantic differential scale)• Yes, No, Don’t Know