Direct Manipulation

Shneiderman and Plaisant, Chapter 6(from slides at http://wps.aw.com/aw_shneider_dtui_5/)

Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct Manipulation interfaces. Human-Computer Interaction, 1, 311-338.

Overview

• Introduction– Big picture first, details and a theoretical account later

• Examples of Direct-Manipulation systems • Explanations of Direct Manipulation

– Hutchins et al. paper

 • Virtual Environments, or Immersive Interfaces

Recall, Interaction Styles from Shneiderman

• Styles for “giving commands” to the system

• 5 main types (p. 67):– Direct Manipulation

• More detail later– Menu selection– Form fillin– Command language– Natural language

• May blend, especially when users are diverse

Paradigm Example: File System Direct Manipulation vs. Command Line

• Direct Manipulation: – User operates on model world to change task world

• Will sharpen this up with more detailed accounts– Dix et al. and Hutchins, Hollan, & Norman (reading)

“Direct Manipulation” Interface to CarNo Computer, but “Direct” – “an ultimate example”

Task: Maneuver the car from Point A to Point B (within certain constraints) 

Interface: Steering wheel, pedals, etc. 

Geographic area through which user (driver) maneuvers car is displayed directly in front of user

(and in high resolution, and in large field of view!) 

Manipulating steering wheel top to left (and right) causes car front to move left (and right)visual and kinetic feedback is immediate

 Pushing brake pedal results in car slowing

arbitrary mapping of user action to car maneuver (automatized)visual and kinetic feedback is immediate

 Definition, again:

User operates on a visually displayed “world” to perform tasks in a domain 

Here, the world on which the user operates, is the real world

For computing tasks, there is the task world and an interface model worldThe user operates on the model world to change the task world

Recall (or not), from 1st class

Interaction Frameworks (Dix)• Interaction:

– Communication between user and system

• Why have a framework?– Allows “precision” in

accounting for differences• E.g., gulfs of execution and

evaluation

– Presents global view• All elements receive

attention

Task

“wor

k on

task

”

“comm

ands”

System

User

“gives”“per

form

s”

“feedback”

Physical System

Goals

Gulf of Execution

Gulf of Evaluation

Background: Models in UI DesignFrom Dix et al.

• Models of UI design consider:

– System– Interface– User

• Will “rewrite” Dix’s account to incorporate standard account of models

Task

“wor

k on

task

” “comm

ands”

System

User

“gives”“per

form

s”

“feedback”and its representation

Background: Models in UI DesignFrom Dix et al.

• Models of UI design consider:

– System– Interface– User

• Will “rewrite” Dix’s account to incorporate standard account of models

• First, it is the System that directly provides feedback about the task

Task

“wor

k on

task

”“gives com

mands”

User

feedbackfe

edba

ck

and its representation

System

Task

“wor

k on

task

” “comm

ands”

System

User

“gives”“per

form

s”

“feedback”and its representation

Models in UI Design

• Next, just change notation a bit

• And …

Task

“wor

k on

task

”“gives com

mands”

User

feedbackfe

edba

ck

and its representation

System

Task “work on task” “gives commands”UserSystem

Models in UI Design

• For our purposes, Dix’s System (which also represents task) comprised of two components

• 1. System model– How system actually

works• Implementation model

• 2. Interface model– Model system

presents to user• Manifest model

Task “work on task” “gives commands”UserSystem

System Model

Interface Model

• Finally, 3. User model– How the user thinks

system works– Conceptual model

UserModel

User/Conceptual/Mental … Model

• How the user thinks system works– Conceptual model

• May or may not correspond to the system!

• Ideally, should match closely, … if (and only if) system reflects task

• Mental model of the system (and task)

Example of DM: Word Processing

• People used to use script files for formatting, until 1980’s

– E.g., nroff

• “word processors” allowed direct manipulation of document (text and images) output

– Albeit with a fair amount of pointing, selecting, and menu use

Example of DM: VisiCalc and Later

• People used to use paper and pencil for accounting, until 80’s

• Shneiderman– The VisiCalc spreadsheet

and its descendants– Imagine erasing a thousand

numbers …• “VisiCalc users delighted in

watching the program propagate changes across the screen.”

– In some cases, spatial representations provide a better model of reality

– Successful spatial data-management systems depend on choosing appropriate:

• Icons • Graphical representations • Natural and

comprehensible data layouts

Example of DM: WIMP File System

• Copy, move, rename, delete– But, is it a file cabinet?

Example of DM: Drawings

• Autocad, etc.

• Spatial

Example of DM: Statistical Analysis

• Specify statistical operations by manipulating icons• Similar to “visual programming”

Example of DM: Statistical Analysis

Example of DM: Games

• D & D with voice chat• Doom, etc.• .. And better

Basic Interaction ParadigmIn context of direct manipulation and command language

• Historically, “commands” differ by paradigm

– Switches, cards – Command line– W-Icons-M-P

• Relatively, direct or indirect manipulation of system

– “giving commands to system”

• Cmd lang: Conversational – User has a conversation with

interface, telling it what to do– 3rd person– User tells system what to do

in task• Then system does it

• Direct Manipulation: Direct – No conversation

• User interacts directly with (representation) of task

– 1st person– Operates directly on model

worlds

Task

“wor

k on

task

”“ gives com

mands”

System

User

feedbackfeed

back

“feedback”and its representation

(possibly manipulable)

General Comments about DMSchneiderman’s Historical Account … and Definition

• Shneiderman described in 1974 – will see sharper description

• Positive feelings associated with “good” user interfaces often attributed to “direct manipulation” interfaces:– Mastery of the interface– Competence in performing tasks– Ease in learning the system originally and in assimilating advanced

features– Confidence in the capacity to retain mastery over time– Enjoyment in using the system– “Eagerness to show the system off to novices” <- !– Desire to explore more powerful aspects of the system

• Definition: – User operates on a visually displayed “world” to perform tasks in a

domain – All in all easier, to see examples, then analyze

Direct (and Indirect) Manipulation Interfaces

• “Direct manipulation” user interfaces contrasted with, – well, “indirect manipulation” interface

• Say, command line systems, or card systems, or switch systems, or …• Recall, discussion of history of computing … relative costs, technology available• But, that was a long time ago …

• Paradigm example: File system manipulation– Bottom line, file system in a computer is just a collection of storage elements

orientations– Even idea of “files” is an abstraction– Organization of files in a hierarchy is a further abstraction

• Command line – cp /usr/fowler/presentations/colloquium_98/hypertextnodes.gif

/usr/fowler/lectures/6362/dir_manip/– Tell the system to take one file and make a copy of it and place that copy in the

designated place

– mv hypertextgraph.gif temp_hypertextgraph.gif– Tell the system to “move” one file to another file with a different name (rename)

Conversations and Model Worlds…In context of direct manipulation and command language

Task

“wor

k on

task

”“ gives com

mands”

System

User

feedbackfeed

back

“feedback”and its representation

(possibly manipulable)

• Interface as conversation:– Interface is a language medium

- user and computer have a conversation about an assumed, but not explicitly represented, world

– Interface is intermediary between user and assumed world about which things are said (and in which commands are issued)

• Interface as model world: – Interface is itself a world where

tuser can act and which changes state in response to user actions

– World (domain) of interest is explicitly represented - there is no intermediary between user and world

– Well done, this can create a sense of acting upon the object of the task domain themselves

• feeling of direct engagement

Direct Manipulation “Characterized”Again, … history and fyi

• An orientation– Really, direct manipulation is perhaps best characterized as an “orientation” to interface

style

• “Definition/orientation”, yet again: – User operates on a visually displayed “world” to perform tasks in a domain

• Among the earliest characterizations (Schneiderman, 1982):1. Continuous representation of the object of interest2. Physical actions or labeled button presses instead of complex syntax3. Rapid incremental reversible operations whose impact on the object of interest is

immediately visible

• Benefits (Schneiderman, 1982):1. Novices can learn basic functionality quickly, usually through a demonstration by a more

experienced user2. Experts can work extremely rapidly to carry out a wide range of tasks, even defining new

functions and features3. Knowledgeable intermittent users can retain operational concepts4. Error message are rarely needed (why?)5. Users can see immediately if their actions are furthering their goals, and if not, they can

simply change the direction of their activity

Direct Manipulation “Characterized”Hutchins et al. orientation

• Typical characteristics of direct manipulation interfaces:1. System (task) objects have visual representations2. User interacts with a model world3. No “conversation” about what to do

the user directly manipulates the model world4. Representations that match the user model facilitate feelings of

directness5. Output from one system action is input for another

• Hutchins et al.’s account– Delineates range of interface phenomena that can contribute to feeling

of directness– Assumption is that the feeling of directness results from the

commitment of fewer cognitive resources• Or (put the other way around), the need to commit additional of the user’s

cognitive resources leads to feelings of indirectness– And, so, account is in essence a cognitive account of direct

manipulation interfaces

Hutchins et al. PaperOverview

• Provides a detailed analysis of the elements of direct manipulation

• Again, conversation metaphor and model world metaphor for interfaces

• Two aspects of directness in direct manipulation: – Distance and Engagement– directness = f(distance, engagement)

• Cognitive effort results from gulfs of execution and evaluation– Gulf = f (Mismatch of system elements and user’s task element)

• gulf of execution – match of commands and mechanisms of system with thoughts and goals of user

• gulf of evaluation– extent to which system output displays present good conceptual model of system

that is readily perceived, interpreted, and evaluated

• Model worlds and inter-referential input/output

• Two forms of distance: Semantic and articulatory

• A space of interfaces:– Engagement x Distance from user goals

Two Aspects of Directness in Direct ManipulationDistance and Engagement

directness = f(distance, engagement) • Directness

– The feeling that results from interaction with an interface (direct vs. indirect) • Distance

– “Distance” between user’s thoughts and system’s requirements for using it to perform task

• (e.g., g32, s/edtor/editor - large distance)– Short distance means translation between user’s thoughts and system is simple and

straightforward • (e.g., insert ‘i’, or type over “edtor” with “editor”)

– Describes factors which underlie the generation of the feeling of directness

Two Aspects of Directness in Direct ManipulationDistance and Engagement

directness = f(distance, engagement) • Engagement

– Feeling that the user is directly manipulating the objects of interest• e.g., dragging a subdirectory to another subdirectory vs. mv /usr/bob/*.* /usr/jo/

• Again, conversation metaphor and model world metaphor for interfaces– talking about and describing an operation vs. – performing operation in a (visual) model world (representing the task domain)

directness = f(distance, engagement)Gulfs of Execution and Evaluation

• Distance: Relationship between user’s task and way task can be accomplished via interface

– To enhance directness, • need to minimize cognitive effort required to map interface actions to task actions• i.e., “bridge the gulf of user’s goals and way they are specified in the system”• “gulfs” exist to the extent of the mismatch of user’s task intentions and how system

used to carry out those task intentions

– Gulfs are about 1. making things happen and 2. knowing if happened– 1. gulf of execution

• match of commands and mechanisms of system with thoughts and goals of user

– 2. gulf of evaluation• extent to which system output displays present good conceptual model of system that

is readily perceived, interpreted, and evaluated

Physical System

Goals

Gulf of Execution

Gulf of Evaluation

Directness and Model World Metaphor1 - Cognitive Effort

• directness = (inversely related to, or 1/) amt of cognitive effort to manipulate system

– Less effort, more directness– More effort, less directness– Leading to feelings of “direct

engagement” with the task

• Cognitive effort results from gulf of execution and evaluation

– Explains why systems are relatively easy or hard to use, and suggests how to minimize effort, or, make system easier to learn and use

Physical System

Goals

Gulf of Execution

Gulf of Evaluation

Task

“wor

k on

task

”“com

mands”

System

User

“gives”“per

form

s”

“feedback”

Directness and Model World Metaphor2 – Controlling world vs. interface

• Yet, “direct engagement” describes a qualitative feeling that user is directly engaged with control of objects (of task), rather than the computer or program which controls the task

– e.g., really drawing the drawing vs. moving a cursor which, then, effects a line or fill

• Characterized as feeling of “first-personess” vs. “third-personess”– User is operating on objects of task directly vs. telling someone (the system) to

operate on objects– Latter is surely case in “conversation metaphor” model of user interfaces

• User operates with linguistic structures (i.e., telling interface what to do)vs.

• User him/herself operating directly on the task objects (with no intermediary)

• *** Central metaphor of system is the model world itself - the key to direct manipulation interfaces

Physical System

Goals

Gulf of Execution

Gulf of Evaluation

Model Worlds andInter-referential Input/Output

• Operating in a model world, of course, requires that representations of task (model world) objects be manipulable in a direct, visual way

• Additionally, if user is to carry out a series of operations (effect a series of instructions/commands which change state of system)

– Then necessary that results of one operation be available for further modification– That is, the output of one operation serve as input for another operation

• Term “inter-referential input/output” refers to this characteristic of interface objects

– e.g., files and folders are moved and more moved again, etc.– words and paragraphs in editor undergo further modification (of course)

Task

“wor

k on

task

”“com

mands”

System

User

“gives”“per

form

s”

“feedback”

Two Forms of Distance: Semantic & Articulatorydirectness = f(distance, engagement)

• Term “language” is quite general– Refers in general to mapping (often arbitrary) of sign to

meaning– Interface language - elements which cause change in system

state– Two “dialects”

• Language for input (causing change to occur)• Language for output (evaluating ensuing state of system),

– e.g., in dm are files in new loc vs. result of mv ... (in fact none unless cd and ls)

• Semantic Distance– Relationship of an expression in interface language to what

user wants to say

• Articulatory Distance– Relationship between meaning of expression and its physical

form

Task

“wor

k on

task

”“com

mands”

System

User

“gives”“per

form

s”

“feedback”

GoalsSemanticDistance

Meaning of Expression

Articulatory Distance

Form of Expression

Two Forms of Distance: Semantic & Articulatorydirectness = f(distance, engagement)

• Semantic Distance– Relationship of an expression in interface language to what

user wants to say

• Articulatory Distance– Relationship between meaning of expression and its physical

form

• E.g., dragging a file icon to another location in a file structure is small – has both small semantic (goals) and articulatory (expression (=visual form) distances

– Both relatively large for command langs

• Menu interface might have intermediate or less– Goals can be easy or hard to semantically map (good vs. not

menu structure)– Likely, form of expression (~command) may have large dist.

Task

“wor

k on

task

”“com

mands”

System

User

“gives”“per

form

s”

“feedback”

GoalsSemanticDistance

Meaning of Expression

Articulatory Distance

Form of Expression

Both Articulatory and Semantic Distances in Gulfs of Execution and Evaluation

• Relationships among:– semantic distance– articulatory distance and – gulfs of execution and evaluation

• Recall, menu example

Direct Engagement

• Summing up, “direct engagement” occurs when user experiences direct interactions with the objects in a domain– Again, “first-personness”

• no intermediary or conversation about what to do• Model world represented in interface is the task

– Interface becomes “invisible” or transparent or disappears– Not clear-cut how to create– And these feeling in fact can come from automization (learning) for

many (or any?) interface

• Laurel (1986) suggests direct engagement requires:– Execution and evaluation should exhibit both semantic and articulatory

directness– Input and output languages should be inter-referential

• allows to feel as if manipulating objects of concern– Rapid system response (ideally, 0 or time appropriate for task)– “Unobtrusive” interface

• interface is not to be noticed, if it is, leads to third-person relation

A Space of Interfaces

• Direct manipulation interfaces minimize semantic and articulatory distances and maximize engagement

Interface asconversation

low-levellanguage

high-level language

low-level world

directmanipulation

Interface asmodel world

Large

Small

Dis

tan

ces

from

Use

r G

oals

Engagement

Problems with DM Interfaces

• Graphical interfaces, generally, have universal usability challenges– E.g., for visually impaired

• Uses much screen space for graphical representations of objects– Detail through text may be displaced

• May still have to learn meaning of visual representations– Not all visual representations have “straightforward” meaning

• Visual representation may be misleading– Even with analogy grasped, may still have to learn limitations

• Requires switching between keyboard and mouse

• Small screens with touch interfaces limit interaction

Challenges for DM Interfaces

• Choosing “right”, or useful, interface metaphor (that has a natural visual representation) a challenge …

– What is a good metaphor for air traffic control?– What is the “best” metaphor– … and for whom

• E.g., file cabinet for secretaries, directed acyclic graph for computer scientists

• “Rapid (<100 ms), incremental, reversable” actions using graphic representations can have high processing demands, even by current standards

– E.g., high processing load db systems– Doing this web-based currently a challenge

Summary – Principles of DM(from Shneiderman)

• Principles:

1. Continuous representations of the objects and actions of interest with meaningful visual metaphors

2. Physical actions or presses of labeled buttons, instead of complex syntax

3. Rapid, incremental, reversible actions whose effects on the objects of interest are visible immediately

End

• .

Fyi - Visual Thinking and Icons

Exploiting visual nature of computersCan challenge the first generation of hackers Individual cognitive styles vary:

Verbal vs. visualLinear vs. holistic“Left vs. right - brain”

 Icon defined

An image, picture, or symbol representing a concept (or a small religious object)

Guidelines for icon design– Represent the object or action in a familiar manner – Limit the number of different icons – Make icons stand out from the background – Consider three-dimensional icons – Ensure a selected icon is visible from unselected icons – Design the movement animation – Add detailed information – Explore combinations of icons to create new objects or actions

Five levels of icon design:– Lexical qualities – Syntactics – Semantics – Pragmatics – Dynamics

Fyi - Direct-Manipulation Programming

Visual representations of information make direct-manipulation programming possible in other domains

 Example

Demonstrational programming is when users create macros by simply doing their tasks

 The five challenges of programming in the user interface:

1. Sufficient computational generality 2. Access to the appropriate data structures and operators 3. Ease in programming and editing programs 4. Simplicity in invocation and assignment of arguments 5. Low risk

 Cognitive dimensions framework may help analyzing design issues of

visual programming environments.