mental models, affordances and the livescribe pen and notebook system

7/30/2019 Mental Models, Affordances and the Livescribe Pen and Notebook System

1/10

Running Head: MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM

Mental Models, Affordances and the Livescribe Pen and Notebook System

Armen J. Chakmakjian

Bentley University


2/10


3/10

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM 3

Mental Models, Affordances and the Livescribe Pen and Notebook System

Recognition of things we encounter as humans requires that we retain some amount of

those observations for later use. (Anderson, 1983) The categorical storage of this information,

alternatively called scripts, schemas or frames, are sequences of actions can be retrieved to fill in

missing information as new situations are encountered. (Long, 1989) Spreading activation is a

process that plays a pivotal role in the retrieval of those schemas. (Anderson, 1983).

Those retrieved schemas are can be referred to as either affordances by cognitive

psychologists or mental models by systems designers. As James Gibson points out, affordance is

a word he made up to describe a concept wasnt in the dictionary at that time since was no word

that described the concept that an environment affords things to an animal that could be either

beneficial or unfavorable in result. (Gibson, 1979, p.127) At about the same time, Donald

Norman wrote about mental models, in which a target system and a user interact to inform and

evolve the users cognitive processes to get a workable result in that interaction (Norman, 1983).

This paper starts by describing memory and categorization, mental models and affordances,

and then delves into the mental model of writing with an instrument on a surface. A description

the Livescribe pen and notebook system technology follows, then an analysis of that functionality

in terms of mental models and affordances and the usability of the system. The paper closes with

some recommendations for the Livescribe system to be easier to learn.

Survey of Research

Memory, Categorization, Schemas, and Semantic Networks

Humans have a conceptual system that stores experiences in categorical fashion.

(Barsalou, 2008) However, in order to categorize that stimulus, it first must be perceived and

stored. Anderson proposed that retention of these experiences comes through the compilation of

cognitive units that are recordings of an external event or an internal computation. These units

are transient and will eventually be encoded into a long-term memory structure called a trace.

The ability to recall a trace is subject to the frequency of repeated associations with that original

trace. (Anderson, 1983)

In order to retrieve and use that perceptual data, the brain applies a heuristic in which

new information that is perceived as relevant is associated with existing traces in process is called

Spreading Activation. (Crestani, 1997) Spreading activation was proposed by Quillian when

doing research into simulating memory search and computer comprehension. In his research,

memory is described as a semantic network containing factual assertions about the environment.

(Quillian, 1969) This theory studied search in that network of information, observing when two or

more nodes (concepts) were traversed in that network and an intersection resulted. (Collins &

Loftus, 1975) These paths are being traversed in parallel, and the excitation of related paths or

related nodes is known as semantic priming; it allows for the quicker processing of that


4/10


association. Semantic priming effects arise from the similarity between the new input (the

target) and the network nodes (the prime) that are activated. (Masson, 1995)

Humans respond to the world based on a continual categorization and classification of

memories. The persistent economy of an organism to gather as much information while

conserving its finite resources leads to the concept of categories; categories reduce the infinite

properties and differences among stimuli, the traces, to something that is cognitively manageable.

(Rosch, 1999) From a cognitive point of view, humans interact in a symbolic environment and

not the physical environment presented to them. (Carley & Palmquist, 1992) Familiarity being the

goal, humans relate to new objects by associating them with other physical objects for which they

have an internal representation, or a metaphorfor classification. (Jonassen & Henning, 1996)

Schemas are the categorical storage of this information as sequences of actions that can be

retrieved or modified later. (Long, 1989) The concept of schemas is best understood in a

discussion of affordances and mental models.

Affordances

In his seminal work, The Ecological Approach To Visual Perception, Gibson posits that

animals, including humans, move through a medium (air most of the time, or water if they are

swimming) that contains substances. Substances are perceived by means of their surfaces.

(Gibson, 1979, p.32) Visual perception of a surface relies on processing of light emanating or

being reflected from that surface. The distinguishing properties of color, texture and the like

define the surface. (Gibson, 1979, p24) Jones points out that Gibsons idea that an objects

affordances are perceived separately from an objects properties was not what Gibson originally

thought and his theory evolved in the years of writing and research in this area between 1966 and

1979. (Jones, 2003) Eleanor Gibson attempted to clarify how affordances are learned. In order to

perceive what the affordance is for a particular object, the actor must also understand the

environment in which it is resident, their own activity in that environment, and the ensuing results

as they interact in the environment. This is essentially how babies learn their surroundings with

their minimal exploratory abilities such as seeing and sucking. (Gibson, E., 2000)

Ellis and Tucker point out that once the affordance is perceived, it also carries with it the

range of possible actions, such as a chair being a thing upon which you sit, or a floor being

something on which you stand, or the difference between the uses of a fork and spoon. However,

their research points them to the conclusion that the list of possible actions actually depends on

the users nervous system disposing of the possible uses of the object. (Ellis & Tucker, 2000) In

this way they may be going where Gibson would not, to the idea that perceptual processes known

as mental models championed by Donald Norman.


5/10


Mental Models

A mental model is a construct that combines the previously described schema with a

process for manipulating the information in the schema. (Merrill, 2000) Donald Norman, taking a

systems level approach, describes mental models as the result of a people interacting with a target

system. He notes with that cursory definition several factors need to be considered. These factors

to be considered are the target system itself, the conceptual model of the target system (in other

words the designers viewpoint of how it will be used), the users mental model (which evolves

with interaction with the target) and the scientists conceptual model of what they are observing.

(Norman, 1983) This would imply that all those mental models interact and affect each other.

From this explanation it could be also extrapolated here that natural things, such as a stone, do not

start with a conceptual model (design intent) associated with them until interaction, discounting

for the moment any metaphysical or religious purpose that might be ascribed to them.

Norman also points out that mental models evolve inherently through contact between

user and object and are highly influenced by the nature of that interaction as well as the users

own previous experiences. (Norman, 1986) Keiras and Bovair point out in their research, when

presenting a new device to a user, even a metaphorically familiar one with new features, those

extra features must be presented in such a way that they themselves support inferences about the

specific resulting actions. (Keiras and Bovair, 1984) This is important when redefining or

enhancing the function of the familiar pen and paper system.

The Mental Model Writing with Pen and Paper

Writing, as the creative process, is a set of activities in which the writer takes on a goal of

organizing a set of thinking processes within the network of their goals and composes them on a

surface or task environment. (Flower & Hayes, 1981) This very well describes the mental model

of the tasks associated with achieving the goal of written composition; it also touches on the

affordances in the writers environment. Gibson neatly describes the specific act of writing with a

pen in his theory of affordances:

A hand-held tool of enormous importance is one that, when applied to a surface, leaves

traces and thus affords trace-making. The tool may be a stylus, brush, crayon, pen, or

pencil, but if it marks a surface it can be use to depict and to write, to represent scenes

and to specify words. (Gibson, 1979, p.134)

Description of the Livescribe Pen and Notebook System

The Livescribe Pen and notebook system was introduced in 2008 (Livescribe, Inc.). The writing

capture system was based on the handwriting capture technology invented by the members of the

Anoto AB team, through many antecedent technologies and patents. (Pettersson & Edso, 2003;

Petterrson & Bjrklund, 2006) The Anoto technology allowed pen strokes to be captured on dot

paper. The deeper purpose of this handwriting capture technology was ascribed to inventor


6/10


Christer Fhrus who envisioned in 1996 a high tech pen that could get the paperwork done

more efficiently. (Anoto AB, 2012) Livescribe combined audio recording with Anotos

technologies to allow for the ability to simultaneously record handwriting and synchronize it with

the concurrent sound track in the vicinity of the user. (Margraff & Chisolm, 2008) The most

obvious application of synchronized audio and handwriting is a student taking notes during a

lecture with the associated synchronized audio and replaying it later.

The Livescribe System in Terms of Affordances and Mental Models

In terms of affordances, the pen offers the existing Gibsons trace-making concept

previously quoted in this paper. Consistent with Gibsons theory, as Norman points out, some of

the affordances are not readily, visually apparent, but the feedback in other non-visual ways of

the system may indicate to the user what the function is. (Norman, 1999) Figure 1 is a picture of

the pens interfaces and controls. The not-visually-apparent affordances are the microphone and

speakers, as well as the camera in the tip. These affordances become apparent in conjunction with

the use of the notebook. The power button affords the system to be turned on, but it also has a

secondary function. When pressed and held, it will begin to do an audio recording even with the

paper is not present. In this case the user must learn that function; it seems slightly disconnected

from the mental model of what a power button does.

Figure 1: Livescribe Pen Interface and Controls

The display is a visual affordance that gives the user familiar information such as the

time, as well as the amount of battery power remaining. The mental model of a battery symbol is

a familiar metaphor for a user. It is clearly a metaphor since a real battery doesnt actually drain

power like a water level decreasing in a vessel.

The notebook page, in Figure 2, is an affordance consistent with Gibsons trace-making

model as previously described. What the user does not readily see is the dot field on the paper

that is used by the camera in the tip of the pen to capture the location of pen strokes on the page.

The user would get a sense that the writing was being recorded when they attach the pen to a

computer and upload the information. The desktop software renders a copy of the page with their

exact strokes. Thus the modern mental model of a page being scanned is inferred.

The audio affordance of the pen and notebook system is invoked by a set of controls on

the paper surface. Metaphorically, these controls look like controls on any modern recording

device and operate within that mental model with one exception. Although there are controls for

record, pause and stop functions consistent to the users mental model, the play control is not

embedded cameraLCD display

Power button

Ballpoint ink refill speaker microphone


7/10


available near them as would be expected. That interaction or affordance is invoked by tapping

on any pen stroke. If a recording is available, the user will hear the audio emanating from the pen

speaker. The user will also note that there are other controls on the bottom of the page to control

the speed of playback and playback location. Finally, augmenting the written record while the

audio is playing is another affordance that creates its own mental model. Once the user discovers

this feature, they will have learned that the synchronization of audio and pen stroke can be

enhanced.

Figure 2: Livescribe Notebook page showing interface and controls

Recommendations

The Livescribe Pen and Notebook System has many useful features that are learnable and

somewhat consistent with the metaphors of writing and recording audio. Invoking other non-

standard functions is less obvious. Playback is controlled on the page by either tapping on the pen

stroke or on a percentage gauge near the bottom of the page. This gauge at the bottom disengages

the temporal correspondence between pen stroke and audio because the writing surface offers no

indication or feedback in that regard. The suggestion would be to have the feature reversed so

when the gauge is tapped during audio playback previously invoked by tapping on a pen stroke,

the pen display shows the percent location of the audio playing for that page. That might be more

consistent with the mental model of the action the user might be trying to accomplish.

The function whereby audio can be recorded even when a notebook is not present is

interesting but difficult for someone to learn. This is accomplished by holding down the power

button which results in the pen is automatically put into audio recording mode. Later using


8/10


playback, if a user writes in a notebook the written and auditory records would become

synchronized. This is a useful feature, but one that would be difficult to teach someone without

documentation. The affordance here, synchronization after a non-written recording, has no

parallel or metaphor that would be invoked in order to learn it except after it is attempted. The

closest metaphor would be in the movie industry where a sound track (such as the background

music) is superimposed on video during editing. It might be better to have the pen query the user

through the display and accompanying audio if the pen detects through its camera that the pen cap

was still attached for several seconds after power on asking the user to tap the power button to

invoke that function after power up. If the pen cap were removed, the prompt would be stopped.

However, significant research would be required to come up with more inferable solution.

Conclusion

This paper studied the memory, categorization, schemas, spreading activation, mental

models and affordances. The paper briefly described the Livescribe pen and notebook system

technology, and then described that functionality in terms of mental models and affordances. The

paper closes with observations about features that are difficult to learn because they have no

corresponding metaphor except to the extent that they are learned when the situation occurs to use

them. The recommendation is to remove the playback gauge at the bottom of the page as it is not

particular usable or understandable in the model that the pen presents. The feature in which the

power button held down starting non-attached notebook recording should also be rethought.


9/10


References

Anderson, J. R. (1983). A spreading activation theory of memory.Journal of verbal

learning and verbal behavior, 22(3), 261-295.

Anoto AB. History. (2012). Retrieved from .

Barsalou, L. W. (2008). Cognitive and Neural Contributions to Understanding the

Conceptual System. Current Directions In Psychological Science (Wiley-Blackwell), 17(2), 91-

95. doi:10.1111/j.1467-8721.2008.00555.x

Carley, K., & Palmquist, M. (1992). Extracting, representing, and analyzing mental

models. Social Forces, 70(3), 601-636.

Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic

processing. Psychological review, 82(6), 407.

Crestani, F. (1997). Application of spreading activation techniques in information

retrieval. The Artificial Intelligence Review,11(6), 453-482. doi:

http://dx.doi.org/10.1023/A:1006569829653

Ellis, R., & Tucker, M. (2000). Micro affordance: The potentiation of components of

action by seen objects.British Journal of Psychology, 91(4), 451-471.

Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing.College

composition and communication, 32(4), 365-387.

Gibson, E. J. (2000). Where is the information for affordances?.Ecological

Psychology, 12(1), 53-56.

Gibson, J. J. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston,

1979.

Jonassen, D. H., & Henning, P. (1996). Mental models: Knowledge in the head and

knowledge in the world. In Proceedings of the 1996 international conference on Learning

sciences (pp. 433-438). International Society of the Learning Sciences.

Jones, K. S. (2003). What is an affordance?.Ecological psychology, 15(2), 107-114.

Kieras, D. E., & Bovair, S. (1984). The role of a mental model in learning to operate a

device. Cognitive science, 8(3), 255-273.

Livescribe, Inc. Livescribe FAQ. (2010). Retrieved from .

Long, D. (1989). Second Language Listening Comprehension: A Schema-Theoretic Perspective.

Modern Language Journal, 73(1), 32.

Marggraff, James & Chisolm, A. (2008). Interactive apparatus with recording and

playback capability usable with encoded writing medium. US: Patent 7453447B2.

Masson, M. E. (1995). A distributed memory model of semantic priming.Journal of

Experimental Psychology: Learning, Memory, and Cognition, 21(1), 3.
http://dx.doi.org/10.1023/A:1006569829653http://dx.doi.org/10.1023/A:1006569829653http://dx.doi.org/10.1023/A:1006569829653


10/10


Merrill, M. D. (2000). Knowledge objects and mental models. InAdvanced Learning

Technologies, 2000. IWALT 2000. Proceedings. International Workshop on (pp. 244-246). IEEE.

Norman, D. (1983). Some observations on mentai models.Mental models, 7.

Norman, D. A. (1986). Cognitive engineering. User centered system design, 31-61.

Norman, D. A. (1999). Affordance, conventions, and design. interactions, 6(3), 38-43.

Pettersson, M. & Edso, T. (2003). Determination of a position code. US: Patent 6548768.

Pettersson, M. & Bjrklund, A. (2006). Method and device for decoding a position-

coding pattern. US: Patent 7999798.

Quillian, M. R. (1969). The teachable language comprehender: a simulation program and

theory of language. Communications of the ACM, 12(8), 459-476.

Rosch, E. (1999). Principles of categorization. Concepts: core readings, 189-206.

mental models, affordances and the livescribe pen and notebook system

Documents