1

CMT 4240 week 7

Qualitative Evaluation Methods

2

Objectives

At the end of this lecture you will be able to carry out and analyse the following evaluation methods: Observation verbal protocols cognitive walkthrough

3

Observation

Direct observationrecorded observationsoftware logging

4

Direct observation

Observation in the real world and in the lab researcher makes notes of user behaviourquestions:

where do you position yourself? What do you watch the screen or the user? which user behaviour is relevant? How fast and how much can you write? Do you have to note conversation literally?

5

Individual Exercise Observation

Imagine you are in the science museumYou evaluate the prototype “secrets

behind your screen”The design team of the museum has

asked you to find out how long on average visitors use the exhibit problems that visitors experience when using

the exhibit

6

Exercise Observation

What do you do (step by step)?Which problems do you expect?What are you going to take with you?5 minutes

7

What are you going to take with you?

Notepad and penscardboard with clip(stop-) watchobservation forms...

8

Potential Problems

Where to position yourself? Need to see the user’s actions and his non-verbal behaviour.

User may feel disturbed, if you watch to closely.How to stop the time and to observe potential

usability problems at the same time.Hawthorne effect

9

Potential Problems

Only one pass at the data collection observer has to decide in real time what is

important and what is not observer may miss important data observer may misconceive events or

misunderstand the user observer has to know from the outset what to

look out for

10

Potential Problems

You may get distractedYour handwriting may be illegibleYou may not remember what you meant

with certain wordsYou may jot down too few words

11

So what do you do?

Preparation: make observation sheets

start time, end timeuser age (approximately) and genderactionsproblems

12

What do you do?

Position sideways between exhibit and user so that you can see both

Write down everything as explicitly as possible

conversations need to be written down as literal as possible

Keep to the observable FACTS not your interpretation of them

13

Recorded observation

Advantage: you can repeatedly look at the recordings and

discover new things every time

Disadvantage: technology is fine, if it works... You need electricity, extension leads, several

cameras, tapes, etc. it can be costly

14

Analysing Observation Material

Recordings: make transcripts transcripts and notes should be coded Informal analysis

what are the problems?

Task based analysis how does the user tackle a given task?

Performance based analysis obtain data on predefined performance measures

15

Verbal protocols

Verbal protocols capture the user’s thoughts and spoken observations

Two types of verbal protocols think aloud protocol post event protocol

verbal protocols are often used in combination with video recording

16

Think aloud protocol

The user is being asked to say out loud what he or she think while he or she is carrying out a task

17

Post event protocol

The user is being asked to think out loud after the task has been completed

18

Exercise Think Aloud Protocol:

Get together in pairsOne student is the observer and one

student is the subjectThe observer makes notesThe subject solves the following problem

while thinking out loud: how does the following series continue:

0, 1/3, 2/4, 3/5, 4/6, ?

19

Solution?

20

Analysis of the protocol

Go through your notes and ask the following questions:

How many different attempts to solve the problem can you identify?

How many different problems did your subject encounter?

Which statement in the protocol supports which problems solving activity and which problem encoutered?

21

Cognitive Walkthrough

Evaluation method based on the idea that… the user learns about the interface in an

exploratory way, the user has goals, the user is applying simple means-ends

reasoning.

22

Cognitive Walkthrough

Evaluation by an expert, who goes through a set task while imitating user performance

23

Preparation: We need four things

1. User description including level of experience with computers and any assumptions made by the designer

2. System description including operations and performance (e.g. paper design)

3. Task description specifying the task that the expert has to carry out from users point of view

4. Action sequence describing the system display and the user actions needed to complete the given task. One system display and one user action together are one step.

24

Process of carrying out a Cognitive Walkthrough

Prototype, user description, system description, task description and action list has to be prepared by the designer/developer

Designer gives these document to an expert.The expert reads the descriptions.The expert carries out the task by following the

action list.The expert asks the following questions with

EACH step of the action list:

25

Questions Cognitive Walkthrough

1. Is the next goal clear at this stage?

2. Is the appropriate action obvious?

3. Is it clear that this action leads to the goal?

4. What problems are there in performing the action?

26

Example: the Automatic Nail Painter

The automatic nail painter is a machine with which one can have one’s nails painted.

27

User Description Automatic Nail Painter

The users are assumed to have no prior knowledge of how to operate the Automatic Nail Painter. However, they are assumed to have experience in putting nail varnish on nails manually

28

System Description Automatic Nail Painter

The system allows one finger to be placed into the machine at a time. The user programmes the machine for use by choosing:- the number of fingers to be varnished- the colours of the varnish to be used- the number of coatings to be applied

29

Task Description Automatic Nail Painter

The user must get all ten fingers varnished with one coat of purple varnish. The system is already switched on when the user begins his/her task. The task is completed, when the tenth fingernail is varnished.

30

Action list Automatic Nail Painter

System: System displays ‘How many fingers do you want to varnish?’User action: user selects number 10 by pressing the + or the - button until the appropriate number is shownSystem: System responds by displaying the current number of fingers selected to be varnished.User action: User presses ok when the correct number has been displayed

31

Action list Automatic Nail Painter (cont.)

System: System displays ‘choose the colour of varnish you wish to apply.’User action: User must select the colour purple by choosing from the colour menu. This includes scrolling up and down until s/he finds purple. S/he then has to press the OK button.System: Systems displays ‘how many coats do you want to apply?’User action: User selects the number of coats by pressing the ‘+’ or ‘-’ button until the 1 is shown.

32

Action list Automatic Nail Painter (cont.)

System: System displays the number of fingers to be varnished, the colour of the varnish and the number of coats to be applied. System asks for confirmationUser action: User presses OK to confirmSystem: System displays ‘Press start to begin varnishing.’User action: User presses start buttonetc.

33

Step 1- Question 1 Automatic Nail Painter

From the action list: Step 1System: System displays ‘How many fingers do you

want to varnish?’User action: user selects number 10 by pressing the + or the - button until the appropriate number is shown

Question 1:Is the goal clear at this stage? Not completely.The user is asked to state the number of fingers to be varnished. He could be thinking that the system deals with one hand at a time.

34

Step 1 - Questions 2, 3, and 4Automatic Nail Painter

Question 2:Is the appropriate action obvious? No.The system dos neither indicate that the user has to use the ‘+’ and ‘-’ buttons nor explain their use.

Question 3:Is it clear that this action leads to the goal? YesIf the user presses the ‘+’ and ’-’ button the numbers are clearly displayed.

Question 4: What problems are there in performing the action?

The action is simple to perform once the user has discovered the use of the buttons.

35

Step 2 – Question 1Automatic Nailpainter

?

36

Summary

Direct and recorded ObservationVerbal Protocol

Think aloud protocol

Cognitive Walkthrough Preparation: System description, user

description, task description, Action sequence Process: for each step in the action sequence

four questions to be answered