2

Usability Assessment and

Evaluation

Dr. Franz J. KurfessComputer Science Department

Cal Poly

4© Franz J. Kurfess

Overview Usability of Knowledge

❖ Introduction usability of tools and systems vs. usability of

knowledge

❖Usability Evaluations

❖Usability Frameworks

❖Usability Considerations for Knowledge-Intensive Computer Systems

❖ Important Concepts and Terms

❖Chapter Summary

6© Franz J. Kurfess

Usability EvaluationsUsability Evaluations

6

7© Franz J. Kurfess

Usability Evaluations

❖ formative evaluation done at different stages of development influences the design of the system as it is being

developed relies on quick feedback from users

or other ways to obtain feedback

❖summative evaluation assesses the quality of a finished product no influence during design and development users can evaluate the actual product

8© Franz J. Kurfess

Four evaluation paradigms

❖ ‘quick and dirty’

❖usability testing

❖field studies

❖predictive evaluation

9© Franz J. Kurfess

Quick and dirty

❖ ‘quick & dirty’ evaluation describes a common practice designers informally get feedback from users or

consultants to confirm that their ideas are in-line with users’ needs and are liked.

❖Quick & dirty evaluations can be done any time

❖The emphasis is on fast input to the design process rather than carefully documented findings.

10

© Franz J. Kurfess

Usability Testing

❖ recording the performance of typical users on typical tasks in controlled settings field observations may also be used

❖ users are watched recorded on video their activities are logged

mouse movements, key presses

❖ evaluation calculation of performance times identification of errors explanation why the users did what they did

❖ user satisfaction questionnaires and interviews are used to elicit the opinions of users

11

© Franz J. Kurfess

Field Studies

❖done in natural settings

❖ to understand what users do naturally and how technology impacts them

❖ in product design field studies can be used to- identify opportunities for new technology- determine design requirements - decide how best to introduce new technology- evaluate technology in use

12

© Franz J. Kurfess

Predictive Evaluation

❖experts apply their knowledge of typical users to predict usability problems often guided by heuristics

❖another approach involves theoretical models

❖users need not be present

❖ relatively quick & inexpensive

13

© Franz J. Kurfess

DECIDE: A framework to guide

evaluationDetermine the goals the evaluation addresses.Explore the specific questions to be answered.Choose the evaluation paradigm and techniques to

answer the questions. Identify the practical issues.Decide how to deal with the ethical issues.Evaluate, interpret and present the data.

14

© Franz J. Kurfess

Determine the Goals

❖ high-level goals of the evaluation

❖ stakeholders in the overall system for the specific evaluation

❖ selection of the usability evaluation paradigm probably influenced by the goals

❖ examples of goals Identify the best presentation method for knowledge. Check to ensure that different access methods are consistent. Investigate how technology affects the usage of knowledge. Improve the usability of an existing product, work flow, or

common practice.

15

© Franz J. Kurfess

Explore the Questions

❖questions to be asked during usability evaluations hypothesis (“research question”) questions can be used to clarify aspects of the goals may include questions used during interactions with

evaluation participants

❖examples for questions related to knowledge Why do users need/want to know this? How do users act on this knowledge? What happens if they do their task without this

knowledge? What is the source of this knowledge? Who is

responsible for veracity, maintenance, access control?

16

© Franz J. Kurfess

Choose the Evaluation Paradigm & Techniques

❖ Four evaluation paradigms ‘quick and dirty’ usability testing field studies predictive evaluation

❖ Techniques observing users asking users about their opinions asking experts about their opinions testing the performance of users modeling the task performance of users

17

© Franz J. Kurfess

Identify Practical Issues

❖ selection of users types of users size of participant pool

❖ budget

❖ schedule

❖ evaluators internal/external

❖ facilities and equipment usability lab recording equipment

18

© Franz J. Kurfess

Decide on Ethical Issues

❖ informed consent form

❖participants have a right to know the goals of the study what will happen to the findings privacy of personal information not to be quoted without their agreement leave when they wish be treated politely

19

© Franz J. Kurfess

Evaluate, Interpret and Present Data

❖may depend on the paradigm and techniques used

❖evaluation aspects Reliability: can the study be replicated? Validity: is it measuring what you thought? Biases: is the process creating biases? Scope: can the findings be generalized? Ecological validity: is the environment of the

study influencing it e.g. Hawthorn effect

20

© Franz J. Kurfess

Usability Considerations for Knowledge-Intensive

Systems❖knowledge-intensive tasks and activities

acquisition, organization, manipulation, retrieval, presentation of knowledge

❖knowledge-centric interaction methods text-based, visual, auditory

❖ tools for knowledge management generic categories, specific tools as examples

❖usability measures for knowledge-intensive tasks qualitative / quantitative subjective / objective

21

© Franz J. Kurfess

Activity: Knowledge Usability in Student

Research❖Scenario: A student (team) needs to do

investigate a topic, e.g. to prepare a research paper, or to work on a project.

❖Task: Identify activities, methods, tools, and usability measures for this scenario.

❖Deliverable: A document created with a tool of your choice that presents the knowledge your team collected about knowledge usability to be posted on Blackboard AI Discussion Board

22

© Franz J. Kurfess

Activity Worksheet :Knowledge Usability

❖ Scenario description: describe the sample scenario that serves as the basis for your

investigation

❖ Tasks and Activities: what are the critical tasks and activities related to dealing with

knowledge

❖ Interaction Methods: how doe you interact with the computer to deal with the

knowledge

❖ KM Tools: what are the tools you’re using

❖ Usability Measures how do you measure the usability of the tools and methods

23

© Franz J. Kurfess

Activity Worksheet:Knowledge Usability

❖ Scenario description:

❖ Tasks and Activities:

❖ Interaction Methods:

❖ KM Tools:

❖ Usability Measures

24

© Franz J. Kurfess

Activity Lecture Preparation:

Knowledge Usability ❖ Scenario description:

an instructor needs to prepare a lecture for a Computer Science class the material will be presented to class in a face-to-face session with computer-based presentation tools

❖ Tasks and Activities: determination of the topic

based on course catalog description, extended course outline, textbook, related courses identification of essential concepts and terms acquisition of knowledge about concepts and terms

possibly recursive until the desired level of detail is reached creation of a framework for the arrangement of the concepts

relationship between concepts, in particular dependencies presentation of the knowledge

sequence, method (natural language in spoken or written form, grapic, demo, simulation)

❖ Interaction Methods: search for related material (documents) organization of knowledge (hierarchical and graph-based frameworks)

❖ KM Tools: search (Google, Google Scholar, CiteSeer, IEEE Digital Library, PolyCat, Spotlight) knowledge collection utility (Google Notebook, Zotero, Scrapbook, DevonThink) knowledge organization tools (outliner, concept mapping, ontology editor)

❖ Usability Measures ratio of useful/irrelevant documents (precision, recall) effort to perform an activity or task (time, basic actions, cost, utilization of resources) user satisfaction “pleasantness”

25

© Franz J. Kurfess

Key points

❖An evaluation paradigm is an approach that is influenced by particular theories and philosophies.

❖Five categories of techniques were identified: observing users, asking users, asking experts, user testing, modeling users.

❖An evaluation framework like DECIDE offers some guidance for the practical aspects of usability evaluations

26

© Franz J. Kurfess

Activity: DECIDE Framework

❖ apply the DECIDE framework to your team project in this class specifically to the usability of knowledge distinguish between usability of the tool or system, and

usability of the knowledge the system deals with

Determine the goals the evaluation addresses. Explore the specific questions to be answered. Choose the evaluation paradigm and techniques to answer the

questions. Identify the practical issues. Decide how to deal with the ethical issues. Evaluate, interpret and present the data.

27

© Franz J. Kurfess

Knowledge UtilizationKnowledge Utilization

27

28

© Franz J. Kurfess

What is Knowledge Utilization?

❖no clear definition

❖often used interchangeably with knowledge dissemination knowledge transfer knowledge usage

❖usually assume two aspects distribution of knowledge, information, or products incorporation of conceptual or instrumental use of

knowledge into relevant activities

29

© Franz J. Kurfess

Knowledge Dissemination and

Utilization❖due to a number of factors, existing knowledge is

not used effectively driven by the dissemination side (researchers), rather

than the knowledge use side (practitioners)finding practical applications of knowledge is often left to

potential users lack of coordinated knowledge utilization activities

ad hoc dissemination models, very few attempts at systematic approaches to utilization

cumbersome accessibilityfinding and accessing knowledge has been the domain of

specialists (librarians, consultants)

[NCDDR 1996]

30

© Franz J. Kurfess

Dissemination Types

❖ spread one-way diffusion or distribution of knowledge and

information

❖ choice users actively seek and acquire knowledge from established

or alternative sources users learn about their options

❖ exchange interactions between people multi-directional flow of knowledge and information

❖ implementation technical assistance, training, interpersonal activities

[NCDDR 1996]

31

© Franz J. Kurfess

Extension Model of Knowledge Use

❖ rational, linear conception of the process of knowledge utilization knowledge is packaged and moved from one place to

anotherbased on the assumption that knowledge can be

arranged into definable, useable units that can be transmitted easily

“getting the word out” based on the hope that potential users will hear about

it, and be willing and capable to utilize it does not reflect the use of knowledge in many

situations

[NCDDR 1996]

32

© Franz J. Kurfess

Complex Model of Knowledge Use

❖ the process usually is not rational nor linear complex

multiple sources, multiple media and paths of delivery interdependencies between individual knowledge items context may be critical

transactional may involve transactions between source (expert) and user

dependent on the background of potential users pre-existing knowledge, beliefs, experiences

❖ the user is involved in the usage process problem-solver constructor of a personal knowledge base

[NCDDR 1996]

33

© Franz J. Kurfess

Knowledge Usage

❖conceptual use changes in levels of knowledge, understanding, or

attitude

❖ instrumental use changes in behavior and practice

❖strategic use manipulation of knowledge to attain specific goals

power, profit, political gain

[NCDDR 1996]

34

© Franz J. Kurfess

Knowledge Usage Metaphors

❖“tabula rasa” the learner’s mind is an empty slate upon which people

“in the know” impress knowledge

❖ learner as a sponge soaking up knowledge, largely without filtering or

processing

❖brain as a computer processes information in a systematic fashion as it is

received from outside sources

[NCDDR 1996]

35

© Franz J. Kurfess

Knowledge Use as Learning Process

❖ role of knowledge dynamic set of understandings influenced by its

originators and its users

❖ role of the learner actively filters and shapes knowledge

integration into existing knowledge constructs models of the the environment

explanations to make sense of the world pre-existing (mis-)understandings may have to be

changedthey result in discrepancies of the mental model

[NCDDR 1996]

36

© Franz J. Kurfess

Dimensions of Knowledge Utilization

❖ dissemination source originator of information initiator of dissemination

❖ content new knowledge supporting information

❖ dissemination medium packaging and transmission of knowledge

❖ user person or organization to receive and apply the

knowledge[NCDDR 1996]

37

© Franz J. Kurfess

Knowledge Life CycleKnowledge Life Cycle

37

38

© Franz J. Kurfess

Knowledge Life Span

[Kaplan 1997]

39

© Franz J. Kurfess

Knowledge Life Cycle

[Kaplan 1997]

40

© Franz J. Kurfess

Utilization of Knowledge Assets

[Konno 2000]

41

© Franz J. Kurfess

Knowledge Usage Issues

❖selection of knowledge

❖composition of knowledge

❖merging of knowledge items

❖modification of knowledge

❖modification of system aspects

❖preservation of consistency

❖user motivation

42

© Franz J. Kurfess

Knowledge Usage Template

[Skyrme 1999]

43

© Franz J. Kurfess

KM Benefits Tree

[Skyrme 1999]

44

© Franz J. Kurfess

Digital LibrariesDigital Libraries

44

[Griffin 2000]

45

© Franz J. Kurfess

Technology vs. Usage

Today’s Technology Centered Systems

User and Usage Centered

Query

Task Context

Organize

ThinkSelect

Evaluate

Output

Collaborate

BrowseTransfer into Search

Gather

Store

Create

Plan

[Griffin 2000]

46

© Franz J. Kurfess

Carnegie Mellon University: Digital Video Libraries• speech, image and natural language

technologies integration

Univ of Michigan: Intelligent Agent Architectures• software agents; resource federation;

artificial service market economies; educational impact

Stanford Univ: Uniform Access• interoperability; protocols & standards;

distributed object architectures; interface design for distributed information retrieval

full-content search and retrieval of video segments

general access, extensibility for heterogeneous distributed resources

new DL cross-disciplinary capabilities, intellectual perspectives and linkages

Project/Research Focus research goal

Digital Libraries Initiative (DLI)Phase 1 Projects

[Griffin 2000]

47

© Franz J. Kurfess

Project/Research Focus

Univ of California, Santa Barbara: Geographic Information Systems

• spatially-indexed data; content-based retrieval; image-compression; metadata

Univ of Illinois: Intelligent Search and the Net• large-scale information retrieval across

knowledge domains; semantic search; SGML; user/usage studies

Univ of California, Berkeley: Media Integration and Access

• new models of “documents”; natural language processing; content-based image retrieval; innovative interface design

resources for geosciences research and education communities

new models and services for multi-media information management in a networked world

semantic retrieval across the net; alternatives for publishers of scientific journals

research goal

Digital Libraries Initiative (DLI)Phase 1 Projects

[Griffin 2000]

48

© Franz J. Kurfess

Computer & Communications CompaniesDigital Equipment CorpXerox CorpXerox PARCIntel CorpApple CorporationBellcoreEastman Kodak CoIBMLockheedInterconnect Tech CorpEnterprise Integration (EIT)BellcoreIntervalMicrosoft CorpBell Atlantic Network ServicesAT&THewlett PackardUnited TechnologiesSoftquadBRS/DatawareSpyglassHitachi

Professional SocietiesAmerican Math Society (AMA)ACMIEEEAmerican Institute of Aeronautics and Astronautics (AIAA)American Physical SocietyAmerican Institute of PhysicsNCGIAAssociation of Research Libraries

Government Agencies and LabsDMA/CIOU S NavyUSGSNASA/ARCRes Agcy of CaliforniaSan Diego Assn of Govts

LibrariesProject Site Univ LibsUSGS LibraryLibrary of CongressCalifornia State LibrarySonoma County LibrarySt. Louis Public LibraryNew York Public Libs

Other UniversitiesSUNY BuffaloUniv of MaineUniv of ArizonaOpen University, U.K.Univ of WisconsinUniv of ColoradoMITCornell Univ

Publishers/Content ProvidersElsevier Science GroupEncyclopedia BritannicaMcGraw-Hill PublishersDialog Information ServicesO'ReillyWAIS IncQED CommunicationsJohn Wiley & SonsU.S. News & World ReportM&T PublishingTribune CompanyUMI

-- DLI Lead Institutions --Carnegie Mellon University of California, Berkeley University of IllinoisStanford University University of Michigan Univ of California, Santa Barbara

Other/Non-ProfitsCNRIEnvironmental Systems Res InstMellon FoundationKellogg FoundationGetty Foundation

Primary & Secondary SchoolsProject-local comm schoolsFairfax County Public SchoolsWinchester-Thurston SchoolAnn Arbor Public SchoolsStuyvesant High School, NYCShasta County Ofc of Edu

Flow of Resources, Technologies, Knowledge, Intellectual Products

International OrgsERCIM

DLI Collaboration and Partnering

[Griffin 2000]

49

© Franz J. Kurfess

PharmaceuticalDesign

StructuralBiology

ChemicalDynamics

Com

pu

ter

Sp

eed

in B

illi

on o

f O

per

atio

ns

per

Sec

ond

1970 1980 1990 2000

0.1

1

10

100

1000

Airfoil

48 Hour Weather

2D PlasmaModeling

72 HourWeather

3D Plasma

Modeling

Vehicle Signature

Climate ModelingFluid TurbulenceHuman Genome

Ocean CirculationQuantum ChromdynamicsSemiconductor ModelingSuperconductor ModelingViscous Fluid Dynamics

Vision and Cognition

Estimate of HiggsBoson Mass

Grand Challenge Requirements

[Griffin 2000]

50

© Franz J. Kurfess

Traffic Requirements for Bandwidth

100

101

102

103

104

105

106

107

108

109

1010

Steady Bursty

CompositeImaging

InteractiveVisualization

VideoTeleconf

TextFile

Transfer

CollaborationTechnology

DistributedComputing

ImageTransfer

Multi-MediaDatabaseAccess

Multi-MediaMail

Electronic Mail

CharacterData Transfer

Ban

dw

idth

Pea

k R

ate

Application Requirements

[Griffin 2000]

51

© Franz J. Kurfess

Computing Capability (FLOPS)

Network Capability(bandwidth)

Two Dimensional Thinking,

Early 1990s...

[Griffin 2000]

52

© Franz J. Kurfess

Computing (flops)Digital content

Com

munic

ati

ons

(ban

dw

idth

, co

nnect

ivit

y)

Digital Libraries technologytrajectory: intellectualaccess to globally distributed information

less more

Three Dimensional Thinking, Mid-90s...

[Griffin 2000]

53

© Franz J. Kurfess

metadatametadata

Today: Context and Structure

[Griffin 2000]

55

© Franz J. Kurfess

Data 01001001100011111100

Information

Knowledge μσ

Understanding The universe is expanding!

cont

ext

anal

ysis

stru

ctur

e

Infe

renc

e

Evolution of Understanding

[Griffin 2000]

56

© Franz J. Kurfess

LifeSciences

2000

PhysicalSciences

Engineering

Life Sciences

Information Sciences

PhysicalSciences

Engineering

2010

Information Sciences

Social Sciences,Humanities

Social Sciences,Humanities

A Vision of Disciplinarity in 2010

[Griffin 2000]

57

© Franz J. Kurfess

FY 1999 Competition Data ~50 proposals requesting $25M ~30 countries Formal Program with UK/JISC (Circular 15/98)

International Digital Libraries Collaborative

Research Program

[Griffin 2000]

58

© Franz J. Kurfess

http://www.euromktg.com/globstats/

April 1999

English 107.2M 56.5%non-English 82.7M 43.5% European 54.9M 30.0%

By end of year 2000

English 160Mnon-English 167M

Languages and the Internet

[Griffin 2000]

59

© Franz J. Kurfess

Stanford InfoBus: CORBA distributed object technology

IC IC

PM

PM

IS

LS LS LSPM

PM

ISIS

IPS

PM

IPS

PM: Protocol Machine

LS: Library Service

IC: Interface Client

IS: Information Source

IPS: Information Processing Service

* objects, collections, services, platforms….

KM Infrastructure

[Griffin 2000]

60

© Franz J. Kurfess

Traditional KM Stress: Service Selection, Organization, Structure for

Access Centralization, Standards Physical objects & standard genres

Contemporary Technological Capabilities (e.g. WWW) Stress:

Flexibility, Openness Rapid Evolution Decentralization (geographic,

administrative) Digital objects, old + new genres

Design Space for KM Systems

& Beyond

Merging Intellectual Perspectives

[Griffin 2000]

61

© Franz J. Kurfess

Audio Level

Key Words

Word Relevance

Camera Motion

Scene Changes

Histogram Scene Analysis

© Carnegie Mellon University 2/96

Application of Integrated Technologies

[Griffin 2000]

62

© Franz J. Kurfess

UC Berkeley Digital Library Testbed

Building Large Collections of Diverse

Information

[Griffin 2000]

63

© Franz J. Kurfess

New Conceptualizations

[Griffin 2000]

64

© Franz J. Kurfess

Goals

❖gather information and build collections conversion of existing collections creation of new collections from scratch integration of collections

❖create new communities communication evaluation collaboration

❖make technology disappear from our awareness and experience

[Griffin 2000]

65

© Franz J. Kurfess

Usage Aspects

❖collection and evaluation of usage information single items vs. sets of items individuals vs. groups temporal relationships info-space relationships

❖examples relevance feedback, user profiles, citation analysis,

hypertext links, collaborative filtering

❖problems technical aspects, quantity of data privacy

66

© Franz J. Kurfess

Post-Test

68

© Franz J. Kurfess

References❖ [Griffin 2000] Stephen M. Griffin, Digital Libraries Initiative, National Science

Foundation, www.dli2.nsf.gov, 2000

❖ [Kaplan 1997] R.M. Kaplan, Knowledge Management - Access to an Untapped Resource, October 1997.

❖ [Konno 2000] Noburu Konno, Knowledge Strategy and “Ba” - The Practice of Knowledge-Based Management. Column, Inc., 2000.

❖ [NCDDR 1996] A Review of the Literature on Dissemination and Knowledge Utilization. National Center for the Dissemination of Disability Research (NCDDR), www.ncddr.org, 1996

❖ [Skyrme 1999].David Skyrme. Knowledge Usage Template. David Skyrme Associates, www.skyrme.com, 1999.

[Gil 2000]