Running User ExperimentsRunning User Experiments

Humeyra Topcu-AltintasHumeyra Topcu-AltintasDavid GirsaultDavid Girsault

Feyi AgaguFeyi Agagu

OutlineOutline

Recruiting ParticipantsRecruiting Participants Selecting and Organizing Tasks to TestSelecting and Organizing Tasks to Test Creating Task ScenariosCreating Task Scenarios Deciding How to Measure UsabilityDeciding How to Measure Usability Prepare Test MaterialsPrepare Test Materials Prepare Testing EnvironmentPrepare Testing Environment Prepare Test TeamPrepare Test Team

Recruiting ParticipantsRecruiting Participants

Finding appropriate Finding appropriate participantsparticipants

Problems about wrong participantProblems about wrong participant More experiencedMore experienced Less experiencedLess experienced

Where should we go?Where should we go? The source, ex: The source, ex: new version of existing os -> customer listnew version of existing os -> customer list

bank teller system -> banksbank teller system -> banks

Should we use company employees?Should we use company employees? Tighter securityTighter security Lower costsLower costs Recruiting is easierRecruiting is easier But, they may not represent the actual usersBut, they may not represent the actual users

Methods to find a Methods to find a participantparticipant

Working with temporary agenciesWorking with temporary agencies AdvertisingAdvertising NetworkingNetworking Working with professional Working with professional

associations associations

Building a database of qualified Building a database of qualified participantsparticipants

For people who might be appropriate For people who might be appropriate for the types of usability testfor the types of usability test

Select appropriate people from db to Select appropriate people from db to recruit for each testrecruit for each test

Reaching and screening Reaching and screening potential participantspotential participants

Sending out letters and Sending out letters and questionnairesquestionnaires

Sending initial letter first, Sending initial letter first, questionnaires to people who questionnaires to people who express interestexpress interest

By telephone By telephone

Introductory letter (to get interest):Introductory letter (to get interest): Type of productType of product The environmentThe environment How long the test will lastHow long the test will last How much we will payHow much we will pay

Screening Questionnaire:Screening Questionnaire: Ask only what we need to knowAsk only what we need to know Ask specific questionsAsk specific questions Ask questions in an order that allows the Ask questions in an order that allows the

recruiter to stoprecruiter to stop

Deciding who should recruit Deciding who should recruit participantsparticipants

Recruiter should be someone whoRecruiter should be someone who understands usability testingunderstands usability testing is enthusiastic about the test is enthusiastic about the test

Planning how many calls it will Planning how many calls it will take to find a participanttake to find a participant

More specialized skills More specialized skills -> More -> More callscalls

More stringent criteria More stringent criteria -> More -> More callscalls

Knowing when to recruit and Knowing when to recruit and schedule participantsschedule participants

A few weeks before the test. A few weeks before the test. Because:Because: people may not know what their people may not know what their

work work schedule will beschedule will be they may forgetthey may forget

Knowing what to say when you Knowing what to say when you are recruitingare recruiting

If it is the first contact, start the recruiting If it is the first contact, start the recruiting with the subjects of the introductory letterwith the subjects of the introductory letter

If the questionnaire has been already filled If the questionnaire has been already filled out, verify the informationout, verify the information

If not, find out quickly if the person If not, find out quickly if the person appropriate or notappropriate or not

Don’t give detailed answers Don’t give detailed answers Don’t assume people show up w/o Don’t assume people show up w/o

remindersreminders

Arranging for payment or other Arranging for payment or other incentivesincentives

If working with a temporary agencyIf working with a temporary agency It will charge for each participantIt will charge for each participant Plus %33 (in general) for themselvesPlus %33 (in general) for themselves

How much should we pay to the participants?How much should we pay to the participants? The one that motivates people to comeThe one that motivates people to come

Not have to be money, maybe gift Not have to be money, maybe gift certificates, travel to the test side, samples certificates, travel to the test side, samples of the product, samples of other products of the product, samples of other products etc.etc.

Making sure that participants Making sure that participants show upshow up

10% no show is common10% no show is common What can we do about the people who forgetWhat can we do about the people who forget

Reschedule themReschedule them What can we do to make sure participants What can we do to make sure participants

come?come? To make the usability test sound interesting, To make the usability test sound interesting,

nonthreateningnonthreatening To confirm the appointment and a day or two To confirm the appointment and a day or two

before the schedule timebefore the schedule time

Selecting and Organizing Selecting and Organizing Tasks to TestTasks to Test

Selecting tasksSelecting tasks

Not possible to test every task users Not possible to test every task users can do with a productcan do with a product

What tasks should we sample?What tasks should we sample? Tasks that probe potential usability Tasks that probe potential usability

problemsproblems Tasks suggested from our concernsTasks suggested from our concerns Tasks derived from other criteriaTasks derived from other criteria Tasks that users will do with the productTasks that users will do with the product

Set a password for the mail account Set a password for the mail account Create a personal mailing listCreate a personal mailing list Move messages between foldersMove messages between folders Sorts items in folderSorts items in folder Read any new mail messageRead any new mail message Read a selected mail messageRead a selected mail message Create and send a mail messageCreate and send a mail message Edit a messageEdit a message File a mail messageFile a mail message Create a folderCreate a folder Forward a messageForward a message Delete a mail messageDelete a mail message Delete a folderDelete a folder Find a mail messageFind a mail message Send a CC of a messageSend a CC of a message Reply to a mail messageReply to a mail message Attach a file to a messageAttach a file to a message Create a distribution listCreate a distribution list

Determining the resources you Determining the resources you need for each taskneed for each task

How long it will take to do the taskHow long it will take to do the task What hardware, software, What hardware, software,

procedures, and other resources will procedures, and other resources will we need to runwe need to run

Creating the final list of tasks Creating the final list of tasks to testto test

After completing task list with their After completing task list with their requirements, eliminate some.requirements, eliminate some.

While eliminating, we should considerWhile eliminating, we should consider The objective of the task, whether we can The objective of the task, whether we can

achieve more than one obj. with itachieve more than one obj. with it Whether a task uses expensive resources is Whether a task uses expensive resources is

worth testingworth testing Whether time consuming tasks are more Whether time consuming tasks are more

important than two or three tasksimportant than two or three tasks

Creating Task ScenariosCreating Task Scenarios

Creating Task ScenariosCreating Task Scenarios

List of tasks for test List of tasks for test presentation presentation to the participantsto the participants

Scenario = embedding tasks in a Scenario = embedding tasks in a short storyshort story

How to create good scenarios?How to create good scenarios?

What is a Scenario?What is a Scenario?

Best defined by example:Best defined by example:““You’ve just bought a new You’ve just bought a new combination telephone and answering combination telephone and answering machine. The box is on the table. Take machine. The box is on the table. Take the product out of the box and set it the product out of the box and set it up so that you can make and receive up so that you can make and receive calls.”calls.”

Makes tasks more realisticMakes tasks more realistic Gives goal, not stepsGives goal, not steps

What Makes a Good What Makes a Good Scenario?Scenario?

Short: no time waste on readingShort: no time waste on reading In the user’s words: easy to understandIn the user’s words: easy to understand Unambiguous: task must be clear Unambiguous: task must be clear

enoughenough Enough information: same data as in a Enough information: same data as in a

real situationreal situation Directly linked to tasks: scenario tests a Directly linked to tasks: scenario tests a

set of tasksset of tasks

Do you Always Give Do you Always Give Participants Written Scenarios?Participants Written Scenarios?

Written scenarios Written scenarios most of the time most of the time

RPG style scenarios RPG style scenarios useful in some useful in some cases (customers, supervisors, cases (customers, supervisors, colleagues…)colleagues…) Written version always necessaryWritten version always necessary

How do you Divide up the How do you Divide up the Tasks and Scenarios for Tasks and Scenarios for

Participants?Participants? Most important criteria = task Most important criteria = task

measurementmeasurement TimeTime Number of errorsNumber of errors Etc.Etc.

Separating tasks Separating tasks separate separate measurementsmeasurements Better precisionBetter precision

How do you Make Participants How do you Make Participants Stop Between Scenarios?Stop Between Scenarios?

Just tell them!Just tell them!

Only provide scenarios one by oneOnly provide scenarios one by one

For written scenarios, ask the user to For written scenarios, ask the user to stop when finished at the end of the stop when finished at the end of the instructionsinstructions

Deciding how to measure Deciding how to measure usabilityusability

Deciding how to Measure Deciding how to Measure UsabilityUsability

Goal: Planning the observations and Goal: Planning the observations and measurements for a usability testmeasurements for a usability test

Quantitative goals:Quantitative goals: vague ideas vague ideas measurable data measurable data

Reasons for planningReasons for planning Focus on important points during testFocus on important points during test Quantitative measuresQuantitative measures

Understanding What You Can Understanding What You Can MeasureMeasure

Performance measures (quantitative)Performance measures (quantitative) Ex: time to finish a task, number of Ex: time to finish a task, number of

wrong menu choiceswrong menu choices Counting keystrokes is difficultCounting keystrokes is difficult

Subjective measuresSubjective measures Quantitative or qualitativeQuantitative or qualitative Ex: user rating the product, collect Ex: user rating the product, collect

spontaneous commentsspontaneous comments

Matching Measures to your Matching Measures to your Goals and ConcernsGoals and Concerns

Performance measures Performance measures quantitative quantitative goalsgoals Ex: time requirement Ex: time requirement time measurement time measurement

Commercial software available for Commercial software available for collecting datacollecting data Taking notes could be as goodTaking notes could be as good

Measuring positive aspects usefulMeasuring positive aspects useful Good aspects of the productGood aspects of the product Rewarding for product developersRewarding for product developers

Matching Measures to the Matching Measures to the Product’s Stage of Product’s Stage of

DevelopmentDevelopment Performance measures adapted to Performance measures adapted to

stage of developmentstage of development Example 1: avoid time Example 1: avoid time

measurements with prototypesmeasurements with prototypes Example 2: avoid measuring usability Example 2: avoid measuring usability

for partially implemented interfacesfor partially implemented interfaces

Setting Quantitative Criteria for Setting Quantitative Criteria for Each Measure and Each TaskEach Measure and Each Task

Must set quantitative criteria for usability Must set quantitative criteria for usability Focus on usersFocus on users Use previous data to set goalsUse previous data to set goals Common senseCommon sense

3 categories of results: excellent, 3 categories of results: excellent, acceptable, unacceptable acceptable, unacceptable value ranges value ranges

Possible criteria modificationPossible criteria modification

Setting Quantitative Criteria Setting Quantitative Criteria (continued)(continued)

Different criteria for each taskDifferent criteria for each task Measures can change with tasksMeasures can change with tasks Small amount of time to get used to Small amount of time to get used to

the environmentthe environment System response time System response time focus on focus on

usersusers

Preparing Test MaterialsPreparing Test Materials

The Concept of minimal RiskThe Concept of minimal Risk

Minimal risk Minimal risk is defined as the is defined as the probability of harm anticipated in the probability of harm anticipated in the test are not greater than what test are not greater than what participant encounters in everyday participant encounters in everyday life.life.

If there is risk involvedIf there is risk involved Eliminate the riskEliminate the risk Follow the federal policies for such testsFollow the federal policies for such tests Don’t do the testDon’t do the test

The Legal FormThe Legal Form

Informed Consent – InformationInformed Consent – Information Explain procedure you will followExplain procedure you will follow Explain the purpose of the testExplain the purpose of the test Explain any risksExplain any risks Explain opportunity to ask questionsExplain opportunity to ask questions

The Legal FormThe Legal Form

Informed consent - ComprehensionInformed consent - Comprehension Convey information clearly and Convey information clearly and completelycompletely Allow questions so participants can Allow questions so participants can make informed decisionmake informed decision If you rush through It seems If you rush through It seems unimportantunimportant

The Legal FormThe Legal Form

Informed consent - VoluntarinessInformed consent - Voluntariness The participant must be free from The participant must be free from

coercioncoercion When participants wish to withdraw When participants wish to withdraw

remain calm and neutral.remain calm and neutral.

The Legal FormThe Legal Form

When obtaining consentWhen obtaining consent Develop a rapportDevelop a rapport Neutral but friendly toneNeutral but friendly tone If video tapping have the camera onIf video tapping have the camera on Body language must not show pressureBody language must not show pressure Be ProfessionalBe Professional Never put participant at riskNever put participant at risk

The Legal FormThe Legal Form

Obtaining consent - Legal issuesObtaining consent - Legal issues Non-disclosure: Participants can’t Non-disclosure: Participants can’t

discuss about product.discuss about product. Waivers: You need participants Waivers: You need participants

permission to use tapes or permission to use tapes or questionnaire.questionnaire.

Participants rights: right to be Participants rights: right to be informed, right to withdraw….informed, right to withdraw….

QuestionnairesQuestionnaires

You need to anticipate questionYou need to anticipate question You need to write questions downYou need to write questions down

So you can ask same questionSo you can ask same question So you will not forget to ask the So you will not forget to ask the

questionquestion Put questions in specific and Put questions in specific and

appropriate formappropriate form

QuestionnairesQuestionnaires

There are several points where you There are several points where you would want to ask participants would want to ask participants questions. They are;questions. They are;

PretestPretest Post-taskPost-task Post-testPost-test

QuestionnairesQuestionnaires

Pretest QuestionnairesPretest Questionnaires

Gather background informationGather background information Verify qualifications of participantsVerify qualifications of participants Questions should be one page or lessQuestions should be one page or less How long have you worked with this How long have you worked with this

product?product?

QuestionnairesQuestionnaires

Post-task QuestionnairesPost-task Questionnaires

To obtain immediate reaction or To obtain immediate reaction or judgments about each taskjudgments about each task

Measure changes in perception over Measure changes in perception over timetime

Did you use the manual to complete the Did you use the manual to complete the task?task?

QuestionnairesQuestionnaires

Post-test QuestionnairePost-test Questionnaire

To gather judgments and rating after To gather judgments and rating after testtest

To clarify any ambiguities that may have To clarify any ambiguities that may have occurredoccurred

General questions and specific questionsGeneral questions and specific questions What did you like least about the product?What did you like least about the product?

Training ScriptsTraining Scripts

To ensure all participants have the To ensure all participants have the same skill levelsame skill level

To provide participants with training To provide participants with training that others do not have.that others do not have.

Task ScenariosTask Scenarios

Package in such a way that it will Package in such a way that it will allow u to observe and record dataallow u to observe and record data

There must be a clear beginning and There must be a clear beginning and a clear ending.a clear ending.

Preparing Testing Preparing Testing EnvironmentEnvironment

Preparing Testing Preparing Testing EnvironmentEnvironment

Arranging equipment and the propsArranging equipment and the props Preparing the productsPreparing the products

Creating sample dataCreating sample data Planning for disasterPlanning for disaster Preparing manualsPreparing manuals

Adjusting the video and audio controlsAdjusting the video and audio controls Preparing data logging materials or Preparing data logging materials or

softwaresoftware Labeling tapes and disksLabeling tapes and disks

Preparing Test TeamPreparing Test Team

Preparing the Test TeamPreparing the Test Team

Number of people to run testNumber of people to run test Three people should be enough!Three people should be enough!

Two usability specialistsTwo usability specialists One member technical staffOne member technical staff

Some go for five but one should Some go for five but one should make sure they are not tripping over make sure they are not tripping over each othereach other

Preparing the Test TeamPreparing the Test Team

Can other people participate in the test?Can other people participate in the test? Observers can view in case they have Observers can view in case they have

any doubts about the test.any doubts about the test.

Should not switch roles during tests?Should not switch roles during tests? No, people should not switch roles No, people should not switch roles

because it might make the results because it might make the results irregularirregular

Preparing the Test TeamPreparing the Test Team

Who should be on the test team?Who should be on the test team?

Testing is done by usability Testing is done by usability professionalprofessional

Testing is done by product developersTesting is done by product developers Testing is done by usability Testing is done by usability

professionals and product developersprofessionals and product developers

Usability specialists as Usability specialists as testerstesters

AdvantagesAdvantages They know how to They know how to

plan and conduct plan and conduct teststests

They know what to They know what to look forlook for

They know how to They know how to analyze dataanalyze data

They become experts They become experts at using lab equip.at using lab equip.

DisadvantagesDisadvantages Not experts in each Not experts in each

productproduct Not experts in how Not experts in how

users for each users for each product do their product do their jobsjobs

Product developers as Product developers as testerstesters

AdvantagesAdvantages They see users They see users

using their productsusing their products They are forced to They are forced to

think about usabilitythink about usability They already know They already know

the productthe product Don’t have to hire Don’t have to hire

peoplepeople

DisadvantageDisadvantage Developers must Developers must

learn new toollearn new tool Developers are not Developers are not

trained observerstrained observers Testing takes time Testing takes time

away from workaway from work The cost may be The cost may be

higherhigher

Preparing the Test TeamPreparing the Test Team

What roles are there for test team members?What roles are there for test team members?

Test AdministratorTest Administrator BrieferBriefer Camera operatorCamera operator Data recorderData recorder Help desk operatorHelp desk operator Product expertProduct expert NarratorNarrator

Preparing the Test TeamPreparing the Test Team

There should be checklists for each roleThere should be checklists for each role Test Administrator - Greet Guest observersTest Administrator - Greet Guest observers Test Administrator – Make sure someone is Test Administrator – Make sure someone is

responsible for setting up for next session.responsible for setting up for next session. Briefer - Check participants nameBriefer - Check participants name Briefer – Let participants see camerasBriefer – Let participants see cameras Product expert – Bring the product backupProduct expert – Bring the product backup

THE ENDTHE END

Questions??? – Science Library!Questions??? – Science Library!