Graceful Interaction In Intelligent Environments

Mikael Wiberg, PhDDepartment of Informatics

Umeå University, 901 87 Umeå, Sweden

mwiberg@informatik.umu.se

+46 (0)90 786 61 15

ABSTRACTThe current trend of blending digital technology into our physical surrounding, and the current attempts made to design for disappearing UIs open up for important questions related to how people will be able to make sense of, and interact with, these new intelligent environments. In this paper we present an empirical interaction study of how people try to make sense of an everyday sensing system that might be a natural element of a more complex intelligent environment in a nearby future. In our study we collected video-recorded data on how 8 subjects explored an interactive lamp (a Mathmos Airswitch interactive lamp). In our study we were able to observe how the subjects quickly developed different personal, technical and dramatic/ affective interaction schemas as ways of exploring the functionality of the interactive lamp and how they then fine-tuned their interaction schemas to effectively interact with the lamp (i.e. switch it on and off and brighten it/dim it). We could also observe how the subjects, in their fine-tuning of their interaction schemas, also adjusted their overall performance in the intelligent environment and how they developed a meta-level interaction schema that we choose to label “graceful interaction”. We discuss our findings and its theoretical and practical implications in relation to design of everyday intelligent environments.

KeywordsAmbient technology, Graceful interaction, Intelligent environments, Interaction schemas, Smart objects, Ubiquitous computing.

1. INTRODUCTIONIn a nearby future our homes, and probably also a lot of public places, are likely to be filled with more or less smart objects that taken together can be viewed as intelligent environments in terms of computationally augmented physical objects designed to do various things and services for us as users of these embedded systems. These smart objects that taken together will constitute larger intelligent environments will also have various

kinds of user interfaces (UIs) so that we, as users in and of these environments, can interact with and control these smart objects.

However, along with the current trend of blending digital technology into our physical surrounding, and along with the current attempts made to design for disappearing UIs, these user interfaces will to a large extent be intentionally visually hidden to the user as this digital technology gets highly interwoven and blended with the physical design. This is the current trend within several emerging areas including e.g. interactive architecture [3,7,8], ambient intelligence, and ubiquitous computing. The “ambient alarm clock” (see figure 1) serves as a good illustration of this current trend. When not switched on the ambient alarm clock appears to be only a piece of wood. However, when touched it switches on and glows just like an ordinary alarm clock. However, the UI is visually hidden to its user and even when switched on the device provide no guidance or interface to its user for e.g. setting the alarm, setting the time, or switching it off again.

Figure 1. The ambient alarm clock. (Source: [8]).Another illustrative example of a more large scale and more environmentally embedded implementation is the Hello.Wall [14,15,16] (see figure 2), a 1.8-meter-wide by 2-meter-high ambient display with integrated light cells and sensing technology designed to show continuously aesthetic patterns conveying presence and mood parameters [14]. An important aspect of the Hello.Wall installation is that it illustrates how the technology is designed to mainly serve as a background technology, i.e. an ambient technology.

Further on, this installation also illustrates an important social aspect of intelligent environments, i.e. that these intelligent environments might be populated by several persons who are simultaneously interacting with each other while interacting

with various smart objects and ambient displays in an intelligent environment.

Figure 2. The Hello.Wall [14,15,16].

The Hello.Wall shares the same UI design principles as the ambient alarm clock in that the UI is visually hidden to the user and there are no help buttons, help menus, or wizards implemented to help or guide the user in interpreting the representations on the Hello.Wall or in his/her understanding of how to execute system commands in front of this ambient display.

On a more general level, and as stated by Mark Weiser [19,20], who originally envisioned this movement towards embedded computational power in everyday objects under the label “ubiquitous computing”, this is in fact one of the ultimate goals with this current movement, i.e. to actually hide away the user interfaces so good that it will be hard for the user to even know that a smart object has a user interface:

“For ubiquitous computing one of the ultimate goals is to design technology so pervasive that it disappears into the surrounding [19,20].”

While this goal has the advantage of not cluttering up our surrounding with all various kinds of graphical user interfaces and buttons on every single smart object that will be part of e.g. a future smart home environment it does at the same time present us with a delicate problem of how we as users of these systems should be able to make sense of various smart objects or “sensing systems” [2] that will surround us. As formulated by Bellotti, et al [2] in their paper “Making sense of sensing systems” these sensing systems/ambient technologies present us with the following five questions:

- When I address a system, how does it know I am addressing it?

- When I ask a system to do something how do I know it is attending?

- When I issue a command, how does the system know what is relates to?

- How do I know the system understands my command and is correctly executing my intended action?

- How do I recover from mistakes?

While all these five questions as formulated by [2] are highly relevant to address when designing ubiquitous information

environments, the list of questions can be further extended if not only considering the system view, but also the user’s view of the system. These questions might then also include e.g. how the user should be able to e.g. find out, understand, and be able to explore what kind of services and functionality an intelligent environment can offer to them, and by which means and through which interaction modalities the user is supposed to interact with the system given that the user interface to a large extent will be visually hidden to the user.

In an attempt to address these questions from a user-centered perspective we have in our project explored this sense making issue empirically. Our overall objective in this project has thus been to address the following research questions:

- How do people make sense of sensing systems?,

- How do people go about discovering what a sensing system can do for them?,

- How do people develop ways of controlling a sensing system?

- And, do people develop any more general interaction schemas that they can reuse whenever they need to interact with a sensing system?

The basic motivation for undertaking this kind of empirically driven, open-ended, and user-centered research is at least three folded: First, we need a better understanding of how people go about exploring intelligent environments and smart objects. This is crucial knowledge for every interaction design project aimed at designing usable ambient devices or more complex intelligent environments. Second, we believe that this kind of empirical interaction studies, with an explicit focus on the user’s behavior in intelligent environments, can be valuable for extracting more general design guidelines, and requirements for design of intelligent environments. Finally, in a long-term perspective, we believe that this kind of empirical interaction studies is a valuable and important element in the development of interaction theories capable of describing, analyzing and predicting human interaction in and with intelligent environments.

The rest of the paper is structured as follows: In the next section we present some related work that has inspired us in our own research throughout this project. We then present our empirical study in which we report on our collected data from an experiment in which 8 subjects were set out to explore and interact with a Mathmos Airswitch interactive lamp as a single example of a smart object that might be a natural part of future, and more complex, intelligent environments. We then present our results from this study followed by a discussion of its theoretical and practical implications in relation to design of everyday intelligent environments before concluding the paper.

2. RELATED WORKThe trend towards intelligent environments, pervasive and ubiquitous computing, and smart objects has been well covered in the literature (e.g. [4]). Further on, e.g. Malcolm McCullough [11] describes in his book “Digital ground” how architecture and mobile, ubiquitous and pervasive technologies will be completely blended in a nearby future and how new intelligent environments will be built upon this complete blend of our physical and virtual world.

Another indicator of this trend towards intelligent environments can be found in current research in the area of ubiquitous computing. Here, a strong trend is towards design of computational things for our homes and for individuals in their everyday life. However, while this trend is very promising most of the work we have found is directed towards design of various smart objects and computationally augmented physical appliances rather than reporting on empirical interaction studies of such smart objects in use. One exception is the paper “unremarkable computing” [17] in which the authors reports from an empirical study of people’s everyday routines and its implications for design of ubiquitous information environments. In their paper they report from an empirical study in which they had an explicit focus on understanding what people do in relation to how the technologies and everyday objects that they use becomes highly invisible or unremarkable to the users. The basic motivation behind their research project was to do an empirical study along the idea of designing this technology in a way that makes it truly invisible, i.e. to the extent that it disappears into the periphery of the users attention. While this empirical interaction study present very interesting empirical results in terms of everyday routines in relation to transparency of everyday objects they did at the same time focus upon the disappearance rather than the appearance of this technology to the user. Thus, their did not focus on how people go about discovering and understanding what an intelligent environment can do for them in their everyday life, and how they effortlessly, and with good precision, can interact with different smart objects in such an environment.

Along this line of thought we found very few empirical studies conducted with an explicit focus on how people, in a very open-ended way, and without any further instructions, go about when asked to explore intelligent environments and smart objects. For one exception see e.g. the research conducted by [1] in which they looked into users interaction with sensing systems from the perspectives of expected, sensed, and desired movements. Most of the studies we found focused on experiments with intelligent environments and smart objects in which clear descriptions or a quite explicit set of instructions have been given to the subjects as a introduction to the environment they should interact with. The typical focus reported has been on task-solving and navigation in these environments (e.g. computer augmented museums) and not on how people develop their own ways of interacting with these environments per se. Thus, the purpose with our study is to contribute to the area of ubiquitous computing and intelligent environments with an empirical study with an explicit purpose on understanding how people actually go about when making sense of highly embedded/ambient sensing systems.

3. EMPIRICAL EXPERIMENT WITH MATHMOS AIRSWITCHBelow we describe the overall design of our empirical study.

3.1 Research questionsIn our empirical study we wanted to address the following research questions: How do people make sense of sensing systems?, How do they go about discovering what a sensing system can do for them?, How do they develop ways of

controlling a sensing system? And, do they develop any more general interaction schemas that they can reuse whenever they need to interact with a sensing system?

3.2 MaterialIn order to explore these research questions empirically we decided to set up a simple experiment environment in which persons could be recorded on video while interacting with a smart object.

One idea that we had for this experiment was that we wanted to keep the technology on a fairly simple level, and limit our study to a quite simple smart object that the subjects could fully explore within a rather short period of time. As a result of this we decided to do our experiment with the Mathmos Airswitch interactive lamp [10] (see figure 3).

Simply described, the Mathmos Airswitch interactive lamp has no on/off-switch on the cord like any traditional lamp. Instead, the user is suppose to control the lamp with various hand movements or gestures, e.g. if the user slowly passes his/her hand above the Mathmos Airswitch it will switch on and if the user moves his/her hand higher or lower above the Airswitch it will brighten or dim the light.

Figure 3 illustrates this idea with the different positions of the hand above the lamp and the corresponding brightness provided.

Figure 3. The Mathmos Airswitch interactive lamp used in our experimental study. (Source: [10]).

On a technical level, the interactive lamp consists of a traditional light bulb connected to two photo sensors via a microcontroller. The photo sensors are mounted vertically inside the lamp directed towards the opening on the top of the lamp.

3.3 SubjectsIn our experiment we collected video-recorded data on how 8 subjects explored the interactive lamp and its functionality. All subjects chosen for this experiment are academics in the area of IT. Thus, we accounted them to be very used to traditional computers. We found this aspect to be important since we did not wanted any unfamiliarity with ordinary computers to affect

this study. Table 1 below shows some demographic data of the subjects that took part in this study.

Subject Male/female Age1 Female 29

2 Female 31

3 Female 38

4 Female 30

5 Male 51

6 Male 30

7 Male 50

8 Male 32

Table 1. User demographics.

3.4 ProcedureThe experiment lasted for a maximum of 30 minutes/subject and the only instructions given to each subject was as follows:

- “As you can see, this is not a traditional lamp since it lacks the typical on/off switch on the cord. Therefore, your mission is to explore how you can interact with the lamp and how you can control it (i.e. switch it on and off, dim it and brighten it). You are supposed to freely explore it and just find out how it works. Please let me know when you feel that you have full control over your interaction with the lamp”.

As seen in this transcription no further instructions were given. There was no information given about the photo sensors or that it reacted to gestures or hand movements. Instead, this lack of instructions was part of the experiment design.

After each experiment a follow-up discussion was initiated to collect comments from the subjects concerning their experiences of this and their own reflections on how they had approached this problem of making sense of this rather simple sensing system.

4. PRELIMINARY RESULTSOn a general level, every subject that took part in our study found out how to interact with the lamp within 15 minutes. Table 2 below present a more precise summary of the time spent by each individual from the time the instructions were given to the moment when the subject said that they now had a good understanding of how to interact with the lamp.

On a more detailed level of analysis we could observe how the subjects quickly developed different interaction schemas as ways of exploring the functionality of the lamp and how they then fine-tuned their interaction schemas to effectively interact with the lamp (i.e. switch on and off the lamp and dim it).

We could also observe how the subjects, in their fine-tuning of their interaction schemas, also adjusted their overall performance in the intelligent environment and how they developed a meta-level interaction schema that we choose to label “graceful interaction”.

Subject Minutes spent to get in full control over the interactive lamp

1 12

2 8

3 7

4 9

5 15

6 11

7 10

8 8

Table 2. Summary of time spent in order to find out how to interact with the lamp.

While it was an interesting observation taken in isolation that the subjects rather quickly understood how to interact with the lamp, an even more interesting thing was how the subjects reached this understanding. Below we present three different techniques, or interaction schemas that the subjects applied to build up a good understanding of how the lamp worked and how they could interact with it and control its functionality. The term “interaction schemas” was chosen here as a way of talking about an almost ritualistic behavior that every subject developed in their interaction with the lamp, i.e. they tried out one interaction modality or one interaction technique. They then tested it over and over again to validate if it actually could be a part of the interaction with the lamp and, if it worked, they tried to extend their interaction schema a little bit further until they could both switch the lamp on and off and dim/brighten the lamp using their own developed interaction schema.

Below we present three different kinds of interaction schemas that were possible to identify during this rather limited experiment. These three interaction schemas include 1) personal interaction schemas, 2) technically-driven interaction schemas, and 3) dramatic/affective interaction schemas.

4.1 Personal interaction schemasEvery subject developed their own personal interaction style or interaction schema, but these schemas where not just personal in the sense of being made up by a single individual. Instead, it was very clear that the interaction schema constructed heavily reflected the individual behind the schema. This, through the incorporation of very personal body language into their interaction schema, ways of moving their hands, gestures applied, and the speed chosen for executing a gesture. The gesture speed issue was actually a quite big problem to three of the subjects who executed exactly the same kind of gesture for brightening the lamp as described in the manufactures manual to the lamp but the gesture was executed to fast so the photo sensors inside the lamp did not manage to register it as a valid system command.

The interaction styles applied for interacting with the lamp was further on very personal since it was highly influenced by the subjects prior experiences which influenced what kind of mental models and interaction metaphors that were chosen and tried out

when approaching and exploring the interactive lamp, e.g. as described under the next section on “technically driven interaction schemas” one subject compared the lamp with some toys that react to loud noises. E.g. a noise-controlled toy car can go in another direction when its “driver” makes loud noises by clapping his/her hands.

Another subject in our study thought about the lamp as an active volcano or a fire possible to put out if placing ones hand in direct directly above and in direct contact with the lamp in an attempt to “smother the fire” (see figure 4).

Figure 4. The “smother the fire” interaction technique

Another subject viewed the interactive lamp as a candle and leaned over the lamp in an attempt to “blow out the candle” (dim the light) or blow on it to give it some fresh air (brighten it).

These two subjects interaction techniques were quite different (i.e. putting ones hand over the lamp vs. trying to blow out the “electric candle”) and it also builds upon two different mental models. At the same time these two examples illustrates something of rather great importance here. The thing is that both of these two techniques work. The first technique works since the distance between the photo-sensor and the hand is rather short, thus resulting in a dimming of the lamp. Also, the second technique worked quite well. Not because of the actual attempts to blow out the candle, but, since the subject placed his/her head above the light and then lean towards it in order to blow it out the photo-sensor interpreted this head movement in the same way as a hand movement which also result in a dimming of the lamp.

On the other hand, these personally made up interaction schemas also presented their creators with some breakdowns in their mental models when they discovered that the interactive lamp did not corresponded 100% to their formulated ideas of how it worked. One example of this kind of breakdown was that it was rather confusing for one of the subjects who had sought of the interactive lamp as a candle when it did not started to glow brighter again (like a normal candle) when the hand was removed after an attempt to smother it out by placing his/her directly above and in direct contact with the lamp.

4.2 Technically driven interaction schemasThree of the subjects started out their exploration of how to interact with the lamp from a highly technical point of departure. They lifted up the lamp, looked underneath it, looked inside it, and tried to see where different sensors or buttons

could be placed on the lamp. When they looked from above straight down into the inside of the lamp they could see the two photo sensors placed beside the light bulb. However, no one of these subjects recognized the sensors as photo sensors and could thus not find out what the sensors would react to. However, after having found out that there were in fact sensors placed inside the lamp they tried several different sensor-driven interaction techniques to make the lamp react to their actions. They clapped their hands to see if it reacted to sound or noises (sound sensors), they touched the lamp both on the outside but also putted their fingers down the lamp from the top (touch sensors), they tried to tilt the lamp (tilt sensors/accelerometers) and they tried to warm the lamp with their hands (heat sensors).

One interesting observation made was that while one of the subjects were tilting the lamp the top of the lamp was facing the table resulting in a dimming of the light. This was taken by the subject experimenting with the lamp as a preliminary confirmation that the lamp did in fact react upon tilting. However, after having tried to tilt the lamp a couple of times without managing to dim it again this idea was abandoned.

4.3 Dramatic/affective interaction schemasWe could also observe some quite dramatic, affective or emotionally driven approaches to the exploration of the interactive lamp. E.g. one of the subjects in our experiment started his/her exploration of the lamp very explicitly by smashing his/her hand as hard as he/she could against the table top on which the lamp was placed. This caused the lamp to literary jump up a couple of centimeters above the table, but it did not, as a big disappointment to the person trying out this affective interaction technique, switch the lamp on. When the lamp did not reacted at all to this quite bold gesture the subject started to laugh and then he/she leaned over the lamp to see if it was damaged after this rather bold “attack”. However, the leaning over the lamp switched it on and this in turn resulted in another affective response from the subject. This time, however, the response was calmer and it was almost with a relief in the voice that subject said to the lamp:

–“So, so… that was not so bad… you glow now so everything seems to be ok…”.

This behavior was both personally and technically related. The somewhat bold expressive way of approaching the lamp is natural for a person that can be said to have an extrovert personality. Further on, the choice of smashing his/her hand on the table top was not randomly chosen but was also motivated by the individuals prior experiences of technologies that react to shakings or loud noises. The follow-up discussion with this particular subject also revealed that this person where used to children’s toys (e.g. monster truck toys) that react upon loud noises and the bumping into things, which makes the toy vehicle go in an alternative direction.

This kind of dramatic and very emotionally expressive way of interacting with a sensing system has also been reported by e.g. [13] in their study of how people interacted with a tangible input device called SenToy, a toy-like affective interface to a computer game.

4.4 Graceful interaction in intelligent environmentsA completely different approach to the dramatic one presented above was also possible to identify when observing several of the subjects. After a while, when the subjects felt that that their interaction with the lamp was stable, effortless and straightforward 6 of the 8 subjects in our study started to fine-tune their interaction schemas and their gestures as to make them more graceful, sophisticated, and elegant. This kind of graceful interaction was not primarily done for the purpose of interacting with the smart object in front of them, but the gracefulness was more closely related to their own appearance in the wider intelligent environment they now considered themselves to be part of. As such, the subjects added an aesthetic level to their interaction schemas in that they did not solely wanted to find out a straightforward functional way of controlling the smart object in front of them, but they also wanted themselves to appear rational and elegant in their body movements in relation to the observer of this experiment, the video camera and other persons that might observe them interacting with the smart object. Figure 5 illustrates the hand of a user that even from the beginning of the experiment, i.e. before even knowing how the lamp actually worked, made various very gentle and smooth hand movements up and down along the side of the lamp. This, almost with dignity or at least with some respectfulness as expressed by slow and graceful hand movements.

Figure 5. The smooth/gentle interaction technique

While this particular attempt made to address the lamp did not work this subject continued with this gentle exploration of the lamp until he/she finally understood how to fully control the lamp.

It should be said though that these three categorizes of interaction schemas as identified in our study are not mutually exclusive. Nor are the categorizes general on a statistical level to anyone that would approach an intelligent environment or an ambient device due to the limited number of subjects included in our study and due to the limited complexity of our intelligent environment consisting of only one smart object. On the other hand, the three categorizes presented above cover quite well the different interactional aspects of our empirical data and this blend between these categorizes also illustrate an important aspect of our study on its own, i.e. that the subjects were faced with a very complex task when set to freely explore this interactive lamp in which they had to try out a quite large repertoire of different interaction techniques, while

simultaneously trying to interpret and make sense of their own interaction with the sensing system as to extract functional and valid system commands from their own body movements and gestures. In facing this complex task it was natural to the subjects to approach this task from several different angles, e.g. relying on prior personal experiences of similar objects, trying out various interaction metaphors, approach the task from a technical horizon, and express his/her frustration over this complex task as well as happiness and pride when the system responded accurately to given commands.

5. DISCUSSIONSo, what implications can be drawn from this rather limited study? From our viewpoint we believe that the study has highlighted both theoretical and practical implications, and in this section we will try to address these different dimensions.

From a theoretical viewpoint the current movement towards information technology explicitly designed to be highly embedded in everyday physical objects present us with several issues that so forth has been taken for granted in traditional usability guidelines for interactive system design. One example is the traditional usability concept of user interface visability. Here, any kind of visual user interface is, by purpose, hidden to the user. Further on, and highly related, these smart objects clearly have some affordances [12] that the physical form can communicate quite well (e.g. that the smart object used in our experiment was a lamp that quite certainly would be possible to switch on and off. At the same time, this physical form provided no further guidelines or help functions that could guide the user in figuring out exactly how to gain access to this functionality. Finally, as for today, most of the ambient objects available today are not designed according to any known or traditional interactional conventions [9]. Instead, each user is left rather alone and has to experientially go about and explore how they can get in control.

From a more practical, or design-oriented viewpoint our study suggests that it might be a good idea if not only the physical form of the smart object could reveal its functional affordances, but also if the digital part of the object could reveal its interaction modality affordances and provide the user with some feedback and clues for how to continue a more well-informed exploration of an, although well hidden, user interface.

Another design implication from our study is that it might be a good idea if the intended interaction schema for a smart object was related to a couple of the most obvious interaction metaphors of their intended users. E.g. as our experiment shown the subjects tried to understand the lamp as a fire, a candle, or a volcano. These different associations were probably not randomly chosen by the subjects but instead probably highly related to the fact that they interacted with a lamp with a light bulb that can shine very bright and also become quite warm if the light is on for a couple of minutes. If a design would support a number of such obvious mental models it might be more likely that the user would understand how to interact with it in an even shorter time.

In our experiment we could also notice how every subject that took part in this experimental study approached the lamp by relying on several different senses. They touched the lamp,

lifted it up, tilted it, tried to warm it with their hands, screamed at it, clapped their hands, etc. In a sense they all had a very tangible and sense-based approach in their exploration of the lamp. This rimes well with the following statement made by Gershenfeld [4] in his book “when things start to think”:

“we need to be able to use all of our senses to make sense of the world” [4, p. 93]

What was clear though was that while the subjects were highly involved in their exploration of the interactive lamp there was at the same time something quite paradoxical about the subject’s own immediate interpretations of what actually triggered any system event. In one sense all subjects were highly aware of the fact that some short sequences of body movements triggered a specific system event, e.g. one user discovered that the lamp switched on when she clapped her hands quite hard just above the lamp. Thus, whenever the lamp by mistake was switched off while she was struggling with learning how to control it she returned to this particular sequence for switching the lamp on again. This, we label repetition of successful interaction schemas. On the other hand, while all subjects consciously relied on this repeated interaction technique for going back to a somewhat safe starting point for further exploration of the interactive lamp they were at the same time totally unable to cut out a specific sequence of their own body movements which had triggered a response from the system and reuse such a sequence as a valid system command. We label this the interaction continuum problem, i.e. the users problem of doing in situ extraction of a particular short sequence of his/her own body movements, which has resulted in an unintended system command, from a longer session of bodily interaction, and then immediately adopt this particular sequence as one possible way of issuing a more specific system command. Although the subjects were quite fast in getting a good understanding of how to interact with the lamp we think that this reliance on repeated interaction in combination with the interaction continuum problem prevented them from an even faster learning cycle and in their development of functional interaction schemas.

Another reflection we have had after this study relates to the observed gracefulness of the subjects interaction with the interactive lamp. This gracefulness might be possible to analyze even further e.g. if interpreting the subjects graceful interaction with the lamp from the viewpoint of Goffman’s [5] theories of human public behavior as everyday social performances on a stage with an audience populating the intelligent environment who are watching and judging every action taken by the single individual. Given this perspective, a single individual interacting with a smart object is not isolated, but instead part of a larger social interplay in which it becomes important to the individual to appear rational and aesthetical in front of the system and in front of any other person that might be present in the same intelligent environment.

The term “graceful interaction” has been used before as a general guiding idea for design of dialogue systems and [6] describes that “graceful interaction” (from a system perspective):

“involves dealing appropriately with anything a user happens to say” [6].

Related to the area of intelligent environments, and given a user perspective rather than a system perspective, a slight modification to this paragraph in the form of a paraphrase might be possible to construct as follows:

Graceful interaction in intelligent environments is about: “dealing appropriately with anything a system happens to do”

i.e. to always, whatever the response (or lack of response) from the sensing system is, act in a way that could be interpreted as rational by others while at the same time serve as a system command. According to this idea we define “graceful interaction” as:

“any way in which a person interact with an intelligent environment that is both effective and effortless to him/her while at the same time appearing to be rational and elegant to anyone else observing him or her while interacting with the system”.

Another aspect of the gracefulness in the subject’s interaction schemas has to do with sorting out actual attempts made to execute system commands or address the sensing system from just any other kind of non-reflected hand movements. The acting out of really explicit and graceful hand movements can be a clear signal to both the system and to other persons nearby in that it communicates that this specific gesture is purposeful and directed towards the system.

6. CONCLUSIONIn this paper we have presented an empirical study of how people make sense of an everyday sensing system that might be a natural part of a more complex intelligent environment in a nearby future.

In our empirical study we collected video-recorded data on how 8 subjects explored an interactive lamp. In our study we could observe how the subjects quickly developed different personal, technical and dramatic/affective interaction schemas as different ways of exploring the functionality of the lamp and how they then fine-tuned their interaction schemas to effectively interact with the lamp (i.e. switch on and off the lamp and dim it/brighten it).

In this paper we have further described how we, during our study, also could observe how the subjects, in their fine-tuning of their interaction schemas, also adjusted their overall performance in the intelligent environment and how they developed a meta-level interaction schema that we have chosen to label “graceful interaction”.

Given these empirical observations we have in this paper also pinpointed some theoretical and practical implications in relation to design of everyday intelligent environments.

It should be said though, that the interaction study conducted have some limitations. The study conducted was quite small in that the experiment only included 8 persons. It was also limited in terms of complexity of the intelligent environment in that our study only covered one individual’s interaction with one smart object, i.e. the interactive lamp. While these two factors can be interpreted as a methodological problem we have in our project instead viewed this as a strength in that we have been able to focus on very detailed and particular aspects of these persons different, and highly personal ways of making sense of a sensing

system, and we have been able to look closely into individual interaction schemas and how such schemas develops around one single smart object. As such we believe that our study has been valuable since it has provided us with the insights reported in this paper concerning e.g. various interaction schemas, the value of openness in design of intelligent environment, and knowledge on how the physical form of a smart object can communicate some interaction related affordances to its user.

The next step in our research project will be to take these valuable experiences from this initial interaction study as a point of departure for conducting a bigger comparative study in which we will compare whether there are any cultural differences between interaction schemas in Sweden vs. in the US. A similar experiment as the one reported in this paper will be conducted in which we plan look into questions like e.g., what kind of mental models were used to make sense of the sensing system? Were there any significant differences in how the users approached the system in Sweden vs. in the US?

To address the two limitations to this reported study as discussed above we also plan to extend our study in terms of technical and functional complexity when it comes to the set up of the intelligent environment. Thus, a second direction for our future work includes the realization of a more complex intelligent environment consisting of both several smart objects as well as tight integration between these objects. Here, we are also planning to set up this environment in a public place as to allow for several persons to simultaneously interact with this environment and allow for these persons to observe and affect each other while interacting with the intelligent environment. In more concrete terms this public interactive environment will be done in tight collaboration with two artists and we plan to design some kind of public, multi-user, and gesture-controlled light installation as part of this years Umeå Autumn Light Festival in Umeå, Sweden [18]. In this forthcoming study we will specifically look into questions related to the social development and use of cross-device interaction schemas, and how such schemas are applied, fine-tuned, and exchanged for effective and graceful interaction in and with intelligent environments.

7. ACKNOWLEDGEMENTSThanks are due to the 8 subjects that took part in this experimental study. Thanks are also due to prof. Victor Kaptelinin and prof. Erik Stolterman for their valuable comments on this project.

8. REFERENCES[1] Benford, S., Schnädelbach, H., Koleva, B., Anastasi, R.,

Greenhalgh, C., Rodden, T., Green, J., Ghali, A., Pridmore, T., Gaver, B., Boucher, A., Walker, B., Pennington, S., Schmidt, A., Gellersen, H., Steed, A. (2005) Expected, sensed, and desired: A framework for designing sensing-based interaction, ACM Transactions on Computer-Human Interaction (TOCHI), Volume 12 Issue 1, March 2005

[2] Bellotti, V., et al. (2002) Making sense of sensing systems: Five questions for designers and researchers, In

proceedings of CHI 2002, p. 415-422, Minneapolis, 20-25 April, 2002.

[3] Bullivant, L. (2005) 4dspace: Interactive Architecture, Wiley-Academy, UK.

[4] Gershenfeld, N. (1999). When Things Start to Think, Owl books, New York.

[5] Goffman, E., (1990) Presentation of Self in Everyday Life, Paperback, Penguin Books Ltd, UK, 1990.

[6] Hayes, Philip J., Reddy, D. Raj (1983): Steps Toward Graceful Interaction in Spoken and Written Man-Machine Communication. In International Journal of Man-Machine Studies, 19 (3) p. 231-284.

[7] Inhabitat – Future-forward design for the world you inhabit, http://www.inhabitat.com/

[8] Interactive Architecture dot Org dhttp://www.interactivearchitecture.org/

[9] Lund, A. & Wiberg, M. (2004) Ambient Displays Beyond Conventions, Presented at the "Designing for attention" workshop at HCI 2004, The 18th British HCI Group Annual Conference, Leeds Metropolitan University, UK 6-10.

[10] Mathmos Airswitch, http://www.mathmos.se/

[11] McCullough, M. (2004) Digital ground: Architecture, Pervasive Computing, and Environmental Knowing, The MIT Press, Cambridge.

[12] Norman, D. (1988) The Design of Everyday Things, Currency doubleday, New York.

[13] Paiva, A., Andersson, G., Höök, K., Mourão, D., Costa, M., Martinho, C. (2002) SenToy in FantasyA: Designing an Affective Sympathetic Interface to a Computer Game, Personal and Ubiquitous Computing, Volume 6 Issue 5-6.

[14] Streitz, N., Magerkurth, C., Prante, T., Röcker, C. (2005) From Information Design to Experience Design: Smart Artefacts and the Disappearing Computer, Interactions, July-August, 2005, p. 21-25.

[15] Streitz, N., Prante, T., Röcker, C., Alphen, D., .Magerkurth, C., Stenzel, R., Plewe, D. (2003) Ambient Displays and Mobile Devices for the Creation of Social Architechtural Spaces: Supporting informal communication and social awareness in organizations. In: O´Hara, K. et. al. (Eds.), Public and Situated Displays: Social and Interactional Aspects of Shared Display Technologies, Kluwer. Pp. 387-409.

[16] Streitz, N., Röcker, C., Prante, T., Alphen, D., . Stenzel, R.,Magerkurth, C. (2005) Designing Smart Artifacts for Smart Environments. IEEE Computer, March 2005, p. 41-49.

[17] Tolmie, P., et al. (2002) Unremarkable Computing, In proceedings of CHI 2002, p. 415-422, Minneapolis, 20-25 April, 2002.

[18] Umeå Autumn Light Festival, http://www.umea.se/ kulturfritid/parknatur/umeahostljusumeaautumnlight2005.4.3aa33d98107237db7c67fff1204.html

[19] Weiser, M. (1991). The computer of the 21st century, Scientific American, 265(3):66-75, September 1991.

[20] Weiser, M., Gold, R., Brown, J. S. (1999). The origins of ubiquitous computing research at PARC in the late 1980s. IBM Systems Journal, 38(4):693-696.