Situated Interactions: Agency in Context and Affordances Trevor Hess

University of North Carolina at Charlotte 9201 University City Blvd, Charlotte, NC 28223

ahess5@uncc.edu

ABSTRACT This paper analyzes the use of affordances in prevailing methods of contextual representation. In doing so, it identi-fies a number of key deficiencies in their description. First a number of authors cited do not conform to the original definition of affordances proposed by James J. Gibson in his original writings. Secondly, their proposed use in repre-sentation of context and the literature cited does not pro-vide a complete account of agency that allows affordances to describe intention and motivation. Through this analysis, the paper constrains the ability of affordances to describe context, and seeks to redefine the logic of contextual de-scription.

Author Keywords Context-awareness, Affordances, Representation

General Terms Design, Theory

INTRODUCTION As computational devices pervade the world we live in today, we should consider the ability for computers to be intelligence agents that can be interacted with. While the dream of an artificial intelligence humanoid no longer ex-ists as a research goal, the need to understand people still exists.

Manifest today in the field of context-aware computing, considerable effort has been committed to accurately repre-senting and interpreting the world to imitate human under-standing. Arising through the observation of people and their actions, context serves as the goal. Defined as our position relative to the physical world, the social realm and historical lineage, the ability to computationally understand context would mean the ability to “understand” human be-havior.

This paper takes a critical look at the prevailing solution to contextual interpretation, embodied interaction. Proposed by Paul Dourish in 2001, it responds to interpretive meth-ods proposed by ubiquitous computing researchers. To in-tegrate contextual interpretation into applications, ubiqui-tous computing researchers have taken a static approach to context. Based on location, identity, time and activity, con-text does not change without one of these parameters changing [1]. Instead Dourish proposes an interactive mod-el of context [4]. In this model context is driven by activity, specifically interaction with the physical and social envi-

ronment. The result is a definition of context that allows it to by dynamic and occasioned based on represented behav-iors. By negotiating between multiple people in the same environment, this can also achieve an interpretation of con-text that extends beyond the individual.

While Dourish’s model solves a number of problems re-garding the representation and interpretation of context, his solution, “embodied interaction” [5] does not consider a number of critical writings that fundamentally alter embod-ied interactions to accurately describe context.

THE REPRESENTATION PROBLEM In 1969, John McCarthy and Patrick J. Hayes identified the need for formal representations of the world as a philosoph-ical problem for computational intelligence [9]. Alone, this is not a difficult challenge. A Turing machine, at its core, satisfies the requirements for general intelligence since its representation, 0 and 1, is not tailored to a specific prob-lem. An issue arises, however, when computers are re-quired to imitate human understandings of the world. When imitation is required, a computer must create a numeric representation of the environment. Once created, the repre-sentation must be interpreted to construct an “understand-ing” that is analogous to human perception of the environ-ment.

Today the issue of representation directly affects the field of Human Computer Interaction (HCI) through ubiquitous computing [18]. While the original paper has inspired a number of research directions, a key component to the ubiquitous computing paradigm is context-awareness. Con-text in this case refers to the meaning of the relationship between the individual and their environment. Thus, when computers attempt to represent and interpret human con-text, it is an analogous to the problem raised by McCarthy and Hayes. In both cases, the computer must be able to represent and interpret the world in the same way that peo-ple understand it.

From a computational perspective, context must be ap-proached through a quantitative view of the environment. Information to be interpreted must be collected through sensors, leading to a mathematical, model driven view on context and human behavior [4]. Since sensors will never provide a complete description of the environment due to inherent inaccuracies, research has focused on the devel-

opment of a set of environmental properties that can be interpreted as context.

In the proposal for context-aware computing, Bill Schilit proposed location, the identity of surrounding people, and environmental objects as the identifying characteristics of context [12]. As his proposal focused specifically on the automatic reconfiguration of computer settings, such as automatically connecting to networks, these properties proved sufficient for achieving his goals. When tailored to specific goals, limited, quantitative measurements of the environment can be sufficient, but fall short of a general approach to context.

Gregory Abowd and Anind Dey provide a general approach by formally defining context and context-aware applica-tions [1]. Instead of identifying the goals of awareness, they define context as any information that can be used to identi-fy the situation of a person, place or object. With this in-formation, context-aware applications are those that use context to “provide relevant information and/or services to the user.” To represent context in the computer they identi-fy location, identity, activity and time as primary de-scriptors.

While primary descriptors of context may be useful when trying to achieve specific goals, they directly contradict the opposing, human understanding of context. This under-standing, which must be described through observation results from activity. As individuals interact with their sur-roundings and each other, they create meaning, which is ultimately described as context.

In response to the inability to accurately represent context, two prevailing methods emerged from the computational community. The first, “situated actions” was proposed by Lucy Suchman [17] responds to the rationalist interpreta-tion of human behavior prevalent in early artificial intelli-gence systems. She critiques the computational understand-ing of human behavior that people create rational plans and then perform a set of action steps to achieve goals. Instead of plans, she states that human behavior must be considered in context, adapting to the changing relationship between the individual and their environment. For application de-signers this means that the situation provides a framework for interpretation, but creating this framework requires pri-or knowledge of the environment and the social group that will use the application.

Paul Dourish expands on Suchman’s proposal of situated actions with “embodied interaction.” In this, Dourish ad-dresses the understanding that context is a dynamic and occasioned property created through interaction between the individual and their environment [4]. His proposal states that social and physical interactions have meaning, and by negotiating these meanings, computers can interpret context arising from behavior. In his definition, Dourish defines context in two parts, being in the physical world

and being in the social world. Being, in this case refers to Heidegger’s phenomenological perspective on being-in-the-world [5], which relates to the relationship between the individual and their environment. In his interpretation of the philosophical tradition, Dourish interprets something as phenomenological if it has be ability to create a phenome-non, in this case context. To account for being in the social world, Dourish cites Suchman, whose situated perspective allows the meaning of social interaction to be presumed based on the relationships between people.

To account for being in the physical world, Dourish intro-duces the idea of affordances. In his writings on embodied interaction affordances are defined through three authors. James J. Gibson provides the initial definition of affordanc-es as opportunities to interact with the environment [8]. Coming from psychology, Gibson conceived affordances as a way to explain explain perception of the world. Then Dourish cites Donald Norman, prevalent in the field of in-dustrial design. In Norman’s definition, affordances are perceived opportunities for interaction with the environ-ment and the objects in it [10]. Finally, William Gaver de-scribes the affordances of technology [6]. Through these definitions affordances allow embodied interactions to ac-count for the qualitative understanding of context is ob-served and described. QUANTITATIVE DESCRIPTIONS OF AFFORDANCES With the identification of representation as a key problem in computational interpretation of context, the ability to quantitatively describe situated actions and embodied inter-action should be assessed. The situated perspective of Suchman and social being in embodied interaction requires the presupposition of the situation so it is not available for translation from qualitative to quantitative representation. Embodied interactions, however, describe a relationship between the individual and the environment through af-fordances, both of which can be measured.

Anthony Chemero provides a holistic perspective on af-fordances and whether they are properties of the individual, the environment or both [2]. Affordances describe opportu-nities for interaction they require both an environment whose had properties that can be utilized, and an individual who has the ability to utilize the properties. Since both en-vironment and individual are required, affordances can be seeing as properties of the relationship between the two. To take Gibson’s [8] example flat surface does not afford standing without someone to stand on it.

This relationship, and the ability to quantify affordances are highlighted in a series of studies on the perception that stairs afford climbing. The studies showed that as the height of risers increase, people show a reduced perception of climb-ability. Since the results produced quantifiable results, the studies showed that affordances and perception could be translated into computational representations.

The first of these studies conducted in 1984 [19] presented participants of varying heights a set of stairs and were asked whether or not they thought they could climb them. Results revealed a split in perception between short and tall participants. Intuitively, the perception of climb-ability stopped for shorter participants at a lower riser height than it did for taller participants.

The second of these experiments conducted in 1992 [15] explored the relationship between muscular strength and perceived climb-ability. The study was conducted by pre-senting young and old adults with a set of stairs, asking them which stair they perceived to be the tallest they could climb. The participants then attempted to climb the stair without assistance moving up an increment if successful or down if not successful. The findings produced two results. First they supported the findings of the earlier study, since a large gap existed between the abilities of tall and short participants. Secondly, they extended the findings by show-ing that perception and ability changed with age and strength.

The findings of these studies mean that perception can be quantified. This allows for a direct translation of qualitative descriptions of affordances to the quantitative descriptions needed to integrate embodied interaction with computa-tional systems. Unlike writings in ubiquitous computing, these studies show that given the same environment, per-ceptions between individuals will vary. From the standpoint of embodied interaction, this means that our interactions with the environment are constrained by our perception of affordances. Since affordances still exists without percep-tion [2], they still stand as a viable description of the envi-ronment and our relationship to it.

AFFORDANCES AS CONTEXT

Expanding Gibsonian Affordances Affordances arose with the publication of The Ecological Approach to Perception by James J. Gibson. Rather than approach perception of the environment through the study of vision, prevalent in psychology at the time, affordances look at behavior and perception of the environment by the opportunities offered to the individual [8]. Essentially, Gib-son defines affordances as mere opportunities; they do not invite or encourage behavior. Without the ability to create activity the Gibsonian definition of affordances is not suffi-cient for describing the observed phenomenon of context.

With the publication of The Psychology of Everyday Things, Donald Norman extends Gibson’s original defini-tion of affordances by stating that affordances encourage interaction [10]. In his description, the presence of af-fordances strongly suggests the operation of objects. When looking at a door, a doorknob suggests that we turn it, while a push plate suggests that we push it. By encouraging interaction, affordances gain the ability to create activity.

William Gaver is cited by Dourish to describe the af-fordances of technology. While Gibson describe affordanc-es explicitly in the physical world, Gaver responds to a need in HCI to study perception in virtual and technologic space [6]. When perceiving an affordance in technology, he states that affordances exist relative to the needs of the in-dividual and are perceived once need arises.

To conclude the discussion about affordances and context, we turn to a paper published by Phil Turner stating that affordances can be used as context. In the paper he talks about two types of affordances: simple and complex. Sim-ple affordances refer to those defined by Gibson. Contrary to Gibson’s definition, however, Turner states that af-fordances both invite and demand interaction based on a gestalt understanding of affordances [13]. Complex af-fordances then account for the social and historical compo-nents of context. In Turner’s definition, affordances create interaction. When these interactions are observed, a human description of context can be created.

Problems of Interpretation While Dourish cites Donald Norman’s work in support of his view on affordances, his misses a key publication of Norman’s in 1999. In this paper Norman retracts his origi-nal position on affordances, which does not account for a situation where they are not perceived [11]. When not per-ceived, affordances cannot suggest certain types, or meth-ods of interaction. By reconsidering the perception of af-fordances, Norman’s description of affordances is realigned with Gibson. While affordances suggest a way to interact, they do not encourage behavior.

In the same way that Norman’s pre-retraction position on affordances incorrectly expands Gibson’s original defini-tion so to does Gaver and Turner [14]. Gaver states that affordances present relevant and desirable actions to the individual. This creates a scenario where affordances only exist when they are needed. While this supports Dourish’s use of affordances in the description of context, affordances that only exist when the user needs them have the same invitation quality that Gibson rejects. By stating that af-fordances invite behavior, Turner is subject to the same criticism. Since affordances exist as opportunities for inter-action whether the individual uses them or not [2], posi-tions justified through invitation or automatic perception should be rejected. Without the ability to create or invite observable behavior, the ability for affordances to describe context is severely limited.

Redefined Affordances as Descriptors of Context The incorrect expansion of Gibson’s definition of af-fordances should be sufficient to prove that alone they are not descriptors of context. Affordances cannot account for the social or historical aspects of context, nor do they pos-sess the invitational qualities that would allow them to de-scribe or encourage interaction.

While the inability to encourage behavior damages Dour-ish’s description of embodied interaction, the relationship between the individual and environment still exists. Since context is described by Dourish as emerging from activity, the interaction, or the usage of affordances can still de-scribe context. The primary change to Dourish’s position is the conflation of perception and practice. Without the abil-ity to invite interaction, perception cannot be used to de-scribe context. It is purely a mental activity that does not manifest itself physically or socially.

The mere presence of an affordance does not guarantee a certain behavior. This restricts Dourish’s position on af-fordances, but does not eliminate it. Through interaction, affordances still describe behavior that can be observed and described as context.

THE PROBLEM OF AGENCY Lying at the core of this misinterpretation of affordances is agency. Defined as the freedom or ability for an individual to act and affect the world around them, agency is a key property that Gibson did not describe [3]. Without provid-ing a theory of agency, affordances cannot describe the intention or motivation behind interaction.

A theory of agency is necessary for affordances to explain why people use certain affordances instead of others. Since affordances do not invite behavior, the opportunity they provide cannot account for the meaning that underlies in-teraction. Simply interacting with an object or environment is not inherently meaningful. Consider the affordances of a piece of paper:

“A piece of paper affords equally writing gibberish and sonnets; it affords writing a shopping list or a note to a colleague; it affords making a map; it affords writing noth-ing upon; it affords wadding up and throwing away; it af-fords making paper airplanes; it affords shredding, cutting into pieces, making paper dolls, making a montage, making Mobius strips; it affords pasting on the wall; if affords col-oring green or fingerpainting upon; if affords photocopying and photocopying onto; if affords stapling to other pieces of paper or clipping them into an album; if affords book-marking; it affords wrapping a package or making into an envelope; it affords cleaning the gaps between teeth; it af-fords cleaning up after your dog or wiping your hands; it affords burning; it affords filtering particulates; it quite simply affords all the possible things I can do with it. My behavior is virtually unconstrained by its affordances.” [3]

Since affordances cannot constrain behavior, agency must act as an interpretive mechanism to explain meaning and choice. To resolve the issue of agency, Edward Reed exter-nalizes the meaning of interaction in the same way the Dourish presents meaning through embodiment. He does this by equating affordances with values of the world and meaning with information [16] Through interaction, or the

use of affordances, individuals gain information or meaning from the world.

To explain how individuals chose between the affordances, Reed refers to selective retention theory [7], a process simi-lar to natural selection. As individuals interact with the world they learn which affordances provide the most mean-ing through a process of variation and selection. In this goal-oriented approach to interaction, intention is measured through the selection of an affordance based on prior expe-rience and perception of the most useful affordance.

Keeping Reed’s theory of agency in mind, the ability to invite behavior should be reconsidered. Since affordances are simply opportunities they will not all invite interaction. Given a piece of paper it is impossible to act on all of its affordances. Likewise, the consideration of agency does not guarantee invitation [21]. Rather, an affordance invites be-havior if the individual perceives its importance in achiev-ing a goal.

WHAT IS LEFT OF EMBODIED INTERACTION? Thus far, embodied interaction has been critiqued for ex-tending the definition of affordances without respecting Gibson’s original definition [8] and Reed’s theory of agen-cy [16] has been considered. By dismissing the ability of affordances to describe context, it should be asked, where does this leave embodied interaction?

In the dismissal of Norman and Gaver, affordances lost their ability to invite behavior through presence or percep-tion. While perception alone could be construed as type of interaction with the environment it does not result in an observable interaction that would be needed to describe context.

The consideration of agency, however, is problematic for embodied interaction. When individuals gain the ability to choose between affordances, the meaning of interaction cannot be given a fixed definition. While still embodied, meaning cannot be ascribed without being defined by the individual. In any application, this reliance on the individu-al would be cumbersome, and should not be considered as a viable solution.

Without a reliable way to have the individual assign mean-ing to their interactions we must rely on the situated per-spective provided in the definition of situated actions. In this configuration, the application designer must presume the meaning of interaction. To preserve agency and respect the dynamic quality of context [4], this situated perspective on interaction must not assume it is correct. While the situ-ated perspective may address the general meaning of inter-action with the environment, the possibility always exists that the interpretation of meaning could be wrong.

Consider a person who has the opportunity to interact with a pencil and a piece of paper. While interaction with either of these objects can be detected, the meaning of interaction

cannot be determined. Consider now that we now that per-son is a novel writer. From a situated perspective, where personal history is considered, we can make an assumption about the meaning of interaction, that the person is writing or planning a novel. With the inclusion of agency, however, we cannot be sure that this assumption is correct. Having the opportunity to fold it, write a poem or note on it means that attention of the application designer must turn towards recognizing whether their assumptions are correct. or mak-ing an assumption and reconfiguring functionality in the periphery.

The issue of situated interactions can be seen in the actions of Microsoft Word’s feature, Clippy [20]. While it knew that you were interacting with a virtual piece of paper, it did not have the ability to assume any meaning from simple interaction. Instead it would make a situated assumption based on continued interaction. When an assumption was made, it would appear and ask a question, “It looks like your writing a letter. Would you like help?” In doing so it respected the individuals ability to confirm or deny the meaning of interaction, but it did not consider that its as-sumption would probably be wrong. Instead it concluded that its assumption was correct, leading to frustration on the part of the user.

CONCLUSION This paper has critiqued the use and definition of af-fordances in Paul Dourish’s definition of embodied interac-tion. While he is correct that context is described through interaction, he does not consider agency. This means that meaning cannot be understood as a static property of inter-action. Instead, the individual determines meaning.

This paper has also described how affordances and the per-ception of affordances can be quantified. This allows for a qualitative and quantitative view of affordances and means that affordances can be described and represented in the computer. Ultimately, this strengthens Dourish’s use of affordances as a way of describing people’s perception of the world. Thus, interaction goes a long way towards mak-ing true context-awareness possible for applications de-signers.

While seeing embodied interaction as a viable direction for computational interpretation and negotiation of context, this paper addresses a critical hole in the definition of af-fordances embedded in Dourish’s description of embodied interaction. The authors cited incorrectly expand upon Gib-son’s original definition of affordances, and do not account for an individual’s freedom of choice in the environment.

While this proposal should not be considered a final or ho-listic solution to contextual representation it serves as a way to create context-aware applications. This relies on situated observations of context, and the recognition that assumptions will not always be correct. The solution is not ideal for application designers, but considering the re-

strictions placed on embodied interaction, it may need to serve as a solution until a viable alternative can be pro-posed.

FUTURE WORK While this paper has presented an example of situated in-teraction in Microsoft Word, this example does not engage the physical environment. Future work, therefore, will test the idea of situated interactions through the conceptual analysis of a physical space. The space will be chosen based on the rigor behind its layout, with ideal candidates designed to encourage certain behaviors.

REFERENCES 1. Abowd, Gregory D., Anind K. Dey, Peter J. Brown,

Nigel Davies, Mark Smith, and Pete Steggles. "Towards a better understanding of context and context-awareness." In Handheld and ubiquitous computing, pp. 304-307. Springer Berlin Heidelberg, 1999.

2. Chemero, Anthony. "An outline of a theory of af-fordances." Ecological psychology 15, no. 2 (2003): 181-195.

3. Cutting, James E. "Two ecological perspectives: Gibson vs. Shaw and Turvey." The American Journal of Psy-chology (1982): 199-222.

4. Dourish, Paul. "What we talk about when we talk about context." Personal and ubiquitous computing 8, no. 1 (2004): 19-30.

5. Dourish, Paul. 2001. Where the action is: the founda-tions of embodied interaction. Cambridge, Mass: MIT Press.

6. Gaver, William W. "Technology affordances." In Pro-ceedings of the SIGCHI conference on Human factors in computing systems, pp. 79-84. ACM, 1991.

7. Ghiselin, Michael T. The triumph of the Darwinian method. Courier Dover Publications, 2003.

8. Gibson, James J. 1979. The ecological approach to vis-ual perception. Boston: Houghton Mifflin.

9. McCarthy, John, and Patrick Hayes. Some philosophical problems from the standpoint of artificial intelligence. USA: Stanford University, 1968.

10. Norman, Donald A. The psychology of everyday things. Basic books, 1988.

11. Norman, Donald A. "Affordance, conventions, and de-sign." interactions 6, no. 3 (1999): 38-43.

12. Schilit, Bill, Norman Adams, and Roy Want. "Context-aware computing applications." In Mobile Computing Systems and Applications, 1994. WMCSA 1994. First Workshop on, pp. 85-90. IEEE, 1994.

13. Turner, Phil. "Affordance as context." Interacting with computers 17, no. 6 (2005): 787-800.

14. Kaptelinin, Victor, and Bonnie Nardi. "Affordances in HCI: toward a mediated action perspective." In Pro-ceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 967-976. ACM, 2012.

15. Konczak, Jürgen, Harry J. Meeuwsen, and Marie E. Cress. "Changing affordances in stair climbing: the per-

ception of maximum climbability in young and older adults." Journal of Experimental Psychology: Human Perception and Performance 18, no. 3 (1992): 691.

16. Reed, Edward. 1996. Encountering the world: toward an ecological psychology. New York: Oxford University Press.

17. Suchman, Lucille Alice. 1987. Plans and situated ac-tions: the problem of human-machine communication. Cambridge [Cambridgeshire]: Cambridge University Press.

18. Weiser, Mark. "The computer for the 21st century." Scientific american 265, no. 3 (1991): 94-104.

19. Warren, William H. "Perceiving affordances: visual guidance of stair climbing." Journal of Experimental Psychology: Human Perception and Performance 10, no. 5 (1984): 683.

20. Whitworth, Brian. "Polite computing." Behaviour & Information Technology 24, no. 5 (2005): 353-363.

21. Withagen, Rob, Harjo J. de Poel, Duarte Araújo, and Gert-Jan Pepping. "Affordances can invite behavior: Reconsidering the relationship between affordances and agency." New Ideas in Psychology 30, no. 2 (2012): 250-258.