Unobtrusive user interfaces for ubiquitous computing: a state of the art of thecomputer to human communication

Alistair Doswald

DIUF – Department of InformaticsUniversity of Fribourg, Switzerland

alistair.doswald@unifr.ch

AbstractIn this paper we explore recent research into unob-

trusive user interfaces for ubiquitous computing. How-ever, rather than looking into both input and output meth-ods for ubiquitous computing, this paper will concentrateonly on the output methods, i.e. the means with whichthe systems deliver information to humans. To this end,recent papers that present studies into such communica-tion, as well as papers about actual devices implementingsuch user interfaces are presented and discussed.

Keywords: unobtrusive user interfaces, ubiquitouscomputing, pervasive computing, HCI.

1. Introduction”Calm technology” is the term coined by Mark Weiserand John Seely Brown in [12] to describe the type of in-terface for ubiquitous computing (a term also coined byMark Weiser in 1988) that would allow a system to passinformation to a user without engaging his full attentionin a stressful manner. Of course, Mark Weiser doesn’tdescribe calm technology in this manner, rather he says”Calm technology engages both the center and the pe-riphery of our attention, and in fact moves back and forthbetween the two”. Still Weiser makes it quite clear thatwhile some information requires the full attention of theuser of a system, other information can merely be present,available in a elegant manner when a user wishes for it,without the need to pass by a traditional desktop com-puter. When we talk about ”unobtrusive user interfaces”in this paper,we refer to the idea of calm technology. Theidea of the user interface passing from periphery of ourattention to the centre and back is the centric idea ofunobtrusive user interfaces. Although we use the term”unobtrusive” to describe the interfaces we are interestedin, it is not a term that is universally used to describethem. Thus, in the literature terms like ”natural” [2],”non-intrusive” - ”transparent” [9] or ”ambient” [4] areused to describe what we will call unobtrusive. Possi-bly some authors may even describe their work in otherwords, while meaning unobtrusive user interfaces.

Ubiquitous computing describes computing any-where and everywhere, with many computing devices peruser. For example a modern car may have about 50 mi-croprocessors [8], but most of them do not provide feed-back to the driver. In [9] ubiquitous computing is definedas allowing (some) user mobility and having a transpar-ent user interface (the traditional desktop computer hav-ing none of those features). That paper also provides apractical ontological framework, which is reproduced inFig. 1. What is interesting for us in this figure is not somuch its description of what ubiquitous computing is, somuch as its examples of transparent interfaces and cat-egories that overlap into ubiquitous computing. Thosecategories - smart environments, wearable computers andaugmented reality - served as a base to choose papers ofsuch systems designed to have an unobtrusive user inter-face.

Figure 1: The mobility/transparency matrix.

The emergence of ubiquitous computing leads tocomputers that we interact with either knowingly or un-knowingly. When the interaction is obvious, having theuser focus exclusively on the output from the ubiqui-tous device may defeat its purpose. For example in [1]

the author describes an application of ubiquitous com-puting (providing important information during paraglid-ing) that must have an unobtrusive interface, else it wouldcompromise the primary activity. However, there is atpresent no clear definition of how such unobtrusive inter-faces may be achieved. The purpose of this state of the artis to give some indication of how current research aims tofill that void, as well as present some of the solutions sci-entists have developped or imagined.

Even though it is possible to find many papers de-scribing unobtrusive user interfaces for ubiquitous com-puting, this does not in fact mean that it is a well de-fined and very active area of research. If one were tosearch for unobtrusive and novel means to transmit com-mands to computers, ubiquitous or otherwise, a plethoraof information is available (e.g. at least an entire sec-tion in [3] is dedicated to such research). However,studies on the means with which a computer may trans-mit information to humans are much less common, withmost output devices consisting of a standard screen, al-though often one of a mobile phone or PDA for ubiq-uitous computing [5] [10]. This is probably due to thewealth of information that can be transmitted through vi-sual information, as well as to the familiarity and ubiq-uity of the medium. However, as [12] may suggest,many of the methods to transmit information unobtru-sively are custom-made for a specific function, whichlimits their re-usability, although not their effectiveness.Studies on the theoretical side of unobtrusive user inter-faces (mainly into user acceptance and preferences) forubiquitous computing are rarer still, but of obvious inter-est.

In section 2, we present two such ”theoretical” userstudies of unobtrusive user interfaces, while section 3 isdevoted to practical applications and devices. Finally sec-tion 4 presents our conclusions on the state of the art.

2. User studiesThe first paper [7] presented in this section is a generalstudy on users reaction to the type of non-conventionaluser-interface that we can expect in ubiquitous comput-ing. The second paper [6] is not directly in the context ofubiquitous computing, being more a study of how peoplereact to an output device that is already ubiquitous in itspresence, namely display monitors. It nonetheless pro-vides valubale insights into unobtrusive user interfaces,as will be seen in sub-section 2.2.

2.1. From Everyday Objects to Computational De-vices: Understanding the Science behind UbiquitousComputing Interface Design

This paper was selected as it is one of the few that ac-tually sought the users opinion on the non-conventionalinterfaces that ubiquitous computers may have, and then

draws conclusions in how such interfaces should becrafted. However, even though its findings are not lim-ited to unobtrusiveness, we will focus on that aspect ofthe paper.

The premise of the paper is to record and evaluate thereaction of people to everyday devices that are comput-erised to convey information. In doing so, the authorshope to present how everyday objects may be modified toconvey information in a constructive manner, in their ownwords ”help guide designers as they go through the rigorsof creating new systems”. The everyday objects used inthe paper, shown in Fig. 2, are the following:

• The infoLAMP, which uses the brightness of thelamp to convey information

• The dataFAN, which uses wind speed from the fan

• The hapticCHAIR, which uses vibration from acushion

Figure 2: the infoLAMP, dataFAN and hapticCHAIR.

To contrast the novel interfaces to well known sys-tems, two desktop interfaces were also used to convey thesame information as the devices: a counter and a progressbar.

The study itself was conducted on a population of 50undergraduates familiar or very familiar with computers,although only 10 of them knew what ubiquitous comput-ing was. The subjects were presented with the follow-ing imagined scenarios: the users where asked to monitorwith the five information sources: 1) outdoor temperaturewhile working on the computer, 2) online buddy status forinstant messaging (also while working on the computer),and 3) progress in relation to the amount of time left dur-ing an exam.

The stated purpose of the experiment is to test fourhypotheses about ubiquitous interfaces, namely

• People prefer desktop over ubiquitous interfaces todisplay everyday information.

• People will be more willing to start using ubiqui-tous interfaces if they perceive them as trustworthyand intuitive.

• The effort required to understand information con-veyed by the ubiquitous interfaces inhibits willing-ness to use.

• People who have never heard of ubiquitous com-puting before will be less trusting of and want to beless dependent on ubiquitous computing systems,impacting their willingness to adopt ubiquitous in-terfaces.

Of these, only the first hypothesis interests us, as it is theonly one which directly deals with the unobtrusive na-ture of the interface. Indeed, preference was measuredover certain criteria, such as if the objects usage was easyto learn, if information was transmitted clearly, and mostimportantly for us, if the objects were intrusive. Unfor-tunately, the ubiquitous computing devices were found tobe more intrusive delivering information when comparedto the desktop options, with the hapticCHAIR being theworst, followed by the dataFAN and the infoLAMP beingthe best of the lot. As predicted by the first hypothesis,the users preferred the desktop alternatives to deliver theinformation with the exception of the second scenario,where the infoLAMP was the preferred medium. In thatsituation the lamp was found to provide the necessary in-formation while not having to focus on the desktop.

These results are unfortunately not altogether surpris-ing for two reasons: the first is that the given devices arerather crude, not specifically designed to minimize theirobtrusiveness, while the second is simply that in two ofthe 3 cases, the users attention is already focused on acomputer, which is capable of presenting the informationin an unobtrusive manner. Indeed, the two desktop inter-faces were sufficiently unobtrusive to displace the aug-mented devices, and even in scenario 2 where the theinfoLAMP was preferred, it was, according to the pub-lished results, still considered more intrusive, if not bymuch. The conclusion which may be drawn is that a sameinterface may be considered intrusive in some situationswhile not in others, leading to the authors to conclude thatdifferent objects are only good at conveying certain typesof information, given their properties. This paper alsohighlights the importance of user participation to evaluatethe validity of an interface, as their underlying hypothe-sis may be mistaken. This is covered in more depth in thefollowing paper.

2.2. Overcoming Assumptions and Uncovering Prac-tices: When Does the Public Really Look at PublicDisplays?

As this paper doesn’t directly deal with unobtrusive userinterfaces for ubiquitous computing, we will only give ashort presentation of this paper. This paper, does, how-ever deal with an informational media that is alreadyubiquitous, namely public display monitors. And al-though the paper doesn’t describe unobtrusive interfaces,

it does, however, give insights into how such devices maycapture the attention of a user, especially when the inter-face is absolutely commonplace and must compete withother interfaces.

The authors of the paper chose to observe the realworld interaction between the public and omnipresentmonitors displaying non-critical information in order tofind out, among others, if the assumption that passersbywill engage in public ambient displays is valid. The re-searchers observed the interaction with 48 existing mon-itors in 24 sites within 3 mid- to large-sized cities, eachfor a time of 1 to 3 hours. They sought to observe howexactly the monitors captured the interest of the public,if at all, namely through studying the placement, sizeand content the monitors displayed. Interest was inferredfrom the time passerbys watched the screens. Althoughthe authors of the paper make detailed observations, wewill limit ourselves to explaining their conclusions here.These are:

• Displays are practically not looked at, either not atall or 1 to 2 seconds, very rarely up to 8 seconds.Apparently people make extremely rapid decisionsabout the value and relevance of large display con-tent.

• Displays must be at eye level, anything above, orespecially below, does not attract any attention, nomatter the content.

• Video, especially of an artistic nature attracts moreattention then any other content, although not nec-essarily longer glances, while non-dynamic con-tent like text rarely caught or held the attention.This seems to validate the artistic value of inter-faces as in [4] and the dangling string of [12].

• Other items of interest could draw interest to dis-play monitors if they were level and relativelyclose.

• Monitors in the path of people, especially if theyare forced to face them, have greater chance to at-tract attention.

• Smaller, more ”private” monitors that only one ortwo people could look at would capture the atten-tion longer than a larger screen (if not with greaterfrequency), even if there was a larger screen nearbythat displayed the same content.

The conclusions of this paper give an insight into designprincipals for how unobtrusive interfaces may capture theattention if necessary, or on the contrary how to ”hide” aninterface. Clearly, even large dynamic displays may notoverly intrude on a user as long as they are pervasive andcommonplace enough.

3. Practical applications

This section focuses on existing or planned devices fol-lowing the ubiquitous computing paradigm that presentthemselves as having unobtrusive user interfaces. Thefirst paper we discuss [4] is in the smart home category,presenting the ways with which the energy consumptionmay be gently transmitted to the inhabitants. The sec-ond paper [10] deals with augmented reality, and morespecifically how an augmented world can be considereda user interface. The third and final paper [11] presents awearable computer: a belt with an haptic interface usedto transmit directional data. As mentioned in the intro-duction, these papers were selected as they are examplesfrom three distinctive domains that overlap the domain ofubiquitous computing. Moreover, all three papers detailthe efforts they made on their user interface.

3.1. Perversive approaches to awareness of energyconsumption

This paper falls into the category of smart homes, morespecifically allowing the users of the smart home to mon-itor their own energy consumption through slow technol-ogy. The authors did not develop a unobtrusive user inter-face themselves, having worked on the software solutionto gather the information and then send the informationto display devices. Their system is designed to be highlymodular, with the ability to add different hardware me-ters and output devices as long as they provided interfaceswritten for the framework used for the software system.The software system also provides a mapping componentthat allows the output devices to change in function ofevents in the measuring system, a control system and andatabase interface.

The interest for the state of the art presented in thispaper is that a large section is dedicated to the type ofinterface they imagine to output the energy consumptionin the smart house. The authors point out that usuallythe only output of the electrical status of a house is acounter which is checked once a year. To make residentsof a smart home aware of their consumption, the authorspropose to use ambient information systems, and moreparticularly informative art. The authors believe that thedimensions of artworks (colour, space, form, scale, etc.)can represent different classes of data, so that the infor-mation is hidden. The authors present three different ex-amples of such artworks that could present information:a digital painting, an indoor fountain and furniture withlight effects (Fig. 3). The digital painting (top) is amodification of David Hockney’s A bigger splash, withthe palm trees subtly changing heights to reflect valuesof power consumption. The fountain (middle) reflectscurrent consumption through the intensity of the runningwater, a system which presents gently the information inboth a visual and auditory manner. As for the cabinet

Figure 3: three examples of informative art.

(bottom), the colour of the lighting reflects current con-sumption. The authors point out that similar dedicateddevices already exists, but they believe that by using fur-niture there is no need for additional appliances.

In this paper, all three devices are unobtrusive, de-signed to be a natural part of the environment, and requirefamiliarity to know the nature of the information that theytransmit. In addition, the information they present canonly be read in an approximate manner. However, in thesame manner as the dangling string example of [12], it isnot necessary to have an exact value, and the nature of theinformation is such that it never needs to attract attentionto itself. These characteristics may in fact be intrinsic toinformative art.

3.2. The World as a User Interface: Augmented Re-ality for Ubiquitous Computing

This paper focuses on mobile augmented reality (AR),presenting the authors’ work in creating such systems, aswell as a discussion in the manner through which sophis-

ticated AR systems may be devised. Although AR mayinvolve any of the senses to communicate information tothe user, this paper focuses exclusively on using visual in-formation to transmit data to the user. The first part of thepaper describes the hardware systems developed by theauthors for mobile AR (a backpack computer with a headmounted display and two hand-held systems), as well asvarious hardware systems for user interface and tracking.Later, the paper briefly discusses the XML models thatcan describe the relation between the real-world objectsand the enhancing digital information. At the end of thedocument, the methods and logic with which the XMLinformation may be translated to visualisations using in-put from the real world is discussed.

The part of the paper which interests us is the descrip-tion of the applications and hypothetical examples whichshowcase the user interface. Their method is to inject vir-tual visual information on top of normal vision of the realworld, either by creating completely new virtual objects,or by enhancing existing objects to display extra informa-tion. The authors clearly state that they sought to maketheir user interfaces unobtrusive, however, as we will see,the unobtrusiveness of some of the examples they givecould be contested, while others are more in line with thestated purpose of this state of the art.

The first application is an outdoor navigation guidefor tourism, which provides both navigation cues to guidethe tourist and information about cultural artefacts of in-terest. The navigation works through the means of visualwaypoints attached by a virtual line on the ground (top ofFig. 4), along with simple directional information if theuser cannot see the next waypoint. The user may also addan arrow pointing directly to his destination. The applica-tion also provides cues to the cultural artefacts of interest,indicating them as icons, the user must then simply lookat the artefact to have it outlined and have readable infor-mation displayed (bottom of Fig. 4). The example pro-vided in the figure is perhaps overly showy, but illustrateswell the concept of injecting virtual information into areal world context. One can easily imagine less showy,yet still informative virtual objects being placed to guidethe user.

The second application is an indoor guide, which usesa combination of a complete 3d miniature model of theenvironment with the path to take, along with a virtualpath superposed on the users view to guide him ( Fig. 5).To better guide the user, the next door to take is high-lighted and an arrow shows next door or turn to take.Once again, the example in Fig. 5 is a rather flashy exam-ple of augmented reality, filled with jarring colours. Andagain, one could easily imagine how such a system couldbe more subtle and we will speculate as to why it is not atthe end of the subsection. The third application is an aug-mented library application, highlighting the book whichthe user wishes to take, or outlining the shelf to which a

Figure 4: an AR tourism application.

book must be put back.

Figure 5: an AR indoor navigation system.

The paper also provides examples of how such aug-mented reality may be used for pedestrian navigation andfor a gas utility company. The pedestrian navigationsystem would work as in the first and second applica-tion. Such a system would seamlessly take over fromcar GPS navigation when leaving a building using a headmounted display (such as augmented sunglasses) to di-rect the user to the destination building, highlighting theentrance to use, and if applicable the destinations roomwindow. Within the building, the system directs the userto an elevator, and displays the destinations floor number.Once on the correct floor, the user is directed to the room.With such a system, the user is helped to go to his desti-nation, but he is still free to focus his attention on othertasks, with only a subtle reminder from the AR system.

The gas utility companys example is a little more

complete, again based on the navigation systems fromthe first and second application and taking it further. Theexample is of a worker that has to find and repair a gasleak in the field. The system would guide the user, whileindicating the location of the pipe within a model. Thesystem would highlight access points that lead to pointswhere the worker could perform measurements to find theleak, and once in place, it would also show the plannedexcavation volume and additional structures within it.Those that are above the pipe (and therefore most rele-vant) would be highlighted in bright colours, but renderedtranslucent to enhance the perception of the main struc-ture, while lower structures would be darkened as theyare not important. Dangerous areas would also be spe-cially highlighted and marked with appropriate signals toindicate the nature of the danger. This example illustratesbest how AR can provide information unobtrusively, sub-tly giving greater importance to relevant structures, whiledecreasing the visibility of unimportant ones. Yet in situ-ations such as dangerous areas, the system can still forcethe attention of the user to the dangers (as it should).

In the last two examples the paper describes user in-terfaces that are far more sophisticated and subtle thenthe first three, especially given the examples in Fig. 4and Fig. 5. In the figures the virtual artefacts clearlyclash with the real world and there are at least two possi-ble reasons for this. The first is that the authors wished tomake sure that the reader of the article could distinguishthe virtual objects to show their intent. The second, morelikely, is that the existing technology could not supporta more subtle rendering of virtual objects, either throughlack of processing power on the mobile devices, or dueto the lack of sufficiently advanced software (or maybeboth). Still, the paper earns its place in this state of theart for both showing the state of unobtrusive interfacesin AR, as well as describing the direction in which theywish to take it.

3.3. ActiveBelt: Belt-type Wearable Tactile Displayfor Directional Navigation

Our interest in this paper resides in two points: the first isthe methods that the authors used to make its kinetic in-terface as unobtrusive as possible, and the second is thatthey tested their concept with subjects in order to refinetheir ideas. As such, it provides a solid basis for an ubiq-uitous unobtrusive kinetic user interface.

This paper itself describes a belt augmented with 8vibration motors and a GPS (Fig. 6). Its aim is purely di-rection, guiding the wearer to his destination, or provid-ing other directional information. The authors are of theopinion that such a device provides relatively unobtrusiveinformation, and moreover is suitable for daily use in mo-bile environments. To make sure that the device isn’t toointrusive, the researchers chose to have a variable inten-sity from 33Hz to 77 Hz, which is perceivable but far

Figure 6: the ActiveBelt.

under the band of 200Hz to 250Hz to which human skinis most sensitive. For the navigation, the direction is pro-vided by a specific vibrator, while distance is provided bypulse information of the vibrators (as the user becomesnearer, the intervals become shorter). Application-wise,the authors propose four uses:

• FeelNavi, a simple navigation application

• FeelSense, a location-aware application that vi-brates in the direction of locations that the user hasexpressed an interest (e.g. boutiques or restaurants)

• FeelSeek, an application that detects if the user hasforgotten to take important RFID-enabled objects(e.g. a wallet with an RFID chip), and first activatesall vibrators to warn the user, then guides him to thelost object

• FeelWave, an entertainment application that trans-mits rhythmic vibration in sync with music.

The FeelNavi application was tested by six subjects, aged21 to 30, that had never used the belt before. Throughthe tests, the researchers were able to validate the func-tions of the belt, and receive calibration information forthe length of the vibrations pulses. Other conclusions thatthe authors received from the test subjects are the follow-ing: vibrations on the back are less well perceived than onthe abdomen, four vibrators may be sufficient for direc-tion, and finally that the application should only transmitinformation when the user is lost, as otherwise it is notdesirable.

4. ConclusionAs was suggested in the introduction, papers on this topicare few and far between. This indicates that while apromising avenue of research, work in unobtrusive in-terfaces for ubiquitous computing is not commonplace.The papers presented in this paper confirm this view, asclearly recent user studies only begin to delve into theopinions of users on ubiquitous interfaces. The paperson existing devices also reflect this opinion, as no two,whether those presented here or those not selected, have acommon methodology or an existing pattern to conform

to. Fortunately, some general trends seem to underlinethe papers: the necessity to keep the user interface dis-creet, even invisible when not needed, the idea that itmust be ”natural” or intuitive, or even the concept thatit should be, if possible, artistic. Still, these trends are notclear and have not yet, to our knowledge, been validatedin studies, possibly due to the prototype and exploratorynature of the ubiquitous devices themselves. The finalconclusion of this state of the art is thus that despite thetime since [12] and the work done in ubiquitous com-puting, research into unobtrusive interfaces is still in itsinfancy.

5. References[1] Pascal Bruegger. ubiGlide, an application for glid-

ing activities based on a motion-aware middlewarearchitecture. Master’s thesis, University of Fri-bourg, 2007.

[2] Javier Ruiz del Solar. Personal robots as ubiquitous-multimedial-mobile web interfaces. In Virglio A. F.Almeida and Ricardo A. Baeza-Yates, editors, LA-WEB, pages 120–127. IEEE Computer Society,2007.

[3] Julie A. Jacko (Ed.). Human-Computer Interaction.New Trends: 13th International Conference, part I.Springer, 2009.

[4] Anton J. Fercher, Martin Hitz, and Gerhard Leitner.Perversive approaches to awareness of energy con-sumption. In AmI-Blocks’08: 2nd European Work-shop on Smart Products: Building Blocks of Am-bient Intelligence, pages 3–8. BoD Books on De-mand, 2009.

[5] Dominique Guinard, Sara Streng, and HansGellersen. Relategateways: A user interface forspontaneous mobile interaction with pervasive ser-vices. In Workshop on Mobile Spatial Interaction(MSI 2007), proceedings of CHI 2007 (Interna-tional Conference on Human Factors in ComputingSystems), San Jose, California, USA, May 2007.

[6] Elaine M. Huang, Anna Koster, and Jan Borchers.Overcoming assumptions and uncovering practices:When does the public really look at public dis-plays? In Pervasive ’08: Proceedings of the 6thInternational Conference on Pervasive Computing,pages 228–243, Berlin, Heidelberg, 2008. Springer-Verlag.

[7] Ali Ndiwalana and D. Scott McCrickard. From ev-eryday objects to computational devices: Under-standing the science behind ubiquitous computinginterface design. International Journal of Comput-ing and ICT Research, 2(2):49–61, 2008.

[8] Karim Nice. How car computers work. HowStuff-Works.com, 2001.

[9] D. Salber, A. K. Dey, and G. D. Abowd. Ubiquitouscomputing: Defining an HCI research agenda foran emerging interaction paradigm. Technical report,Georgia Institute of Technology, 1998.

[10] Dieter Schmalstieg and Gerhard Reitmayr. Theworld as a user interface: Augmented realityfor ubiquitous computing. In Georg Gartner,William E. Cartwright, and Michael P. Peterson, ed-itors, Location Based Services and TeleCartogra-phy, Lecture Notes in Geoinformation and Cartog-raphy, pages 369–391. Springer, 2007.

[11] Koji Tsukada and Michiaki Yasumura. Activebelt:Belt-type wearable tactile display for directionalnavigation. In UbiComp 2004: Ubiquitous Com-puting, pages 384–399, 2004.

[12] Mark Weiser and John Seely Brown. Designingcalm technology. PowerGrid Journal, 1, 1996.