Data-collection techniques

Contents

Types of data Observations Event logs Questionnaires Interview

Two types of data

Usability MeasuresUse/problemDescriptions

Observations

Observe in real-time or from video recordings

Can be used both for measures and descriptions

Usability MeasuresUse/problemDescriptions

http://www.mangold.de/english/intoverview.htm

Measures from observations

Particularly suitable for measuring effectivity and efficiency

How many tasks were performed?

How quickly was the task performed?

Were the goals achieved?

How many errors were made?

Effectivity

Efficiency

Use/problem descriptions from observations

Description of breakdowns Describe situations when there is a

”breakdown” in the users interaction with the product

Description of use Describe how the user actually used the

product

Event logs

Logging of the user’s interaction with the product

Typically collected automatically Can be used both for measures and

descriptions

Usability MeasuresUse/problemDescriptions

http://zing.ncsl.nist.gov/WebTools/WebVIP/overview.html

http://zing.ncsl.nist.gov/WebTools/FLUD/example-pretty.html

http://zing.ncsl.nist.gov/WebTools/VisVIP/overview.html

http://zing.ncsl.nist.gov/hfweb/proceedings/etgen-cantor/

http://zing.ncsl.nist.gov/hfweb/proceedings/etgen-cantor/

http://zing.ncsl.nist.gov/hfweb/proceedings/etgen-cantor/

http://www.mangold.de/english/patternexample.htm

http://zing.ncsl.nist.gov/WebTools/VisVIP/overview.html

Measures from event logs

Suitable for measuring effectivity and efficiency

Effectivity

Efficiency

Use/problem descriptions from event logs

Visualise interaction patterns (Quantify use)

Questionnaire

Self-made or validated Most useful for collecting measures

Usability Measures

Self-made or validated?

Designing your own questionnaire allows you to address issues of particular interest; however, it is difficult to create good items

Validated questionnaires (e.g. SUMI) have typically gone through extensive testing. Thus, test items are generally good, but they may not address the issues you are particularly interested in.

http://www.acm.org/~perlman/question.cgi?form=NAU

http://www.acm.org/~perlman/question.cgi?form=ASQ

http://www.acm.org/~perlman/question.cgi?form=PUEU

Measures from questionnaires

Can be used for measuring all attributes, but is particularly suitable for measuring satisfaction; we are interested in the user’s subjective opinion

Satisfaction

Interview

Structured, semi-structured, open Most useful for use/problem

descriptions

Use/problemDescriptions

Use/problem descriptions from interviews

Information about previous experiences and knowledge that might explain the user’s performance

Obtain explanations: the user’s preferences Breakdowns errors

Student tasks

• What data-collection techniques do you think are suitable for testing your product?

• What do you believe are the strengths and weaknesses of the technique?