Proposition of Autonomous Interactive Desk EnvironmentEichi Takaya

Graduate School of Science and Technology,Keio UniversityYokohama, Japanetakaya@keio.jp

Rio WatanabeGraduate School of Informatics and Engineering,

The University of Electro-CommunicationsChofu, Japan

rio.watanabe.394@gmail.com

Keisuke OhnoGraduate School of Science and Technology,

Keio UniversityYokohama, Japankskon@keio.jp

Satoshi KuriharaGraduate School of Science and Technology,

Keio UniversityYokohama, Japansatoshi@keio.jp

ABSTRACTUnder the recent ubiquitous information infrastructure, everyonecan use various information and services anywhere at any time.And as a development of the infrastructure, ambient informationinfrastructure is on the way. On the novel infrastructure, the ambi-ence surrounding people autonomously interacts with them, and itis required to support their lives. In this study, we focused on thesupport of people who do work on a desk at their homes or offices,and proposed the autonomous interactive desk environment (AIDE)system. In AIDE, desktop objects and application windows whichare considered as agents predict tasks from the user’s work log andselect appropriate actions. Specifically, the position of applicationwindows or desktop objects is changed to make it easier for usersto use them. As a result of our preliminary experiments, we showedthe validity of AIDE system in terms of supporting desk works.

CCS CONCEPTS• Human-centered computing→ Ambient intelligence;

KEYWORDSAmbient Intelligence, Human-Agent Interaction

ACM Reference Format:Eichi Takaya, Rio Watanabe, Keisuke Ohno, and Satoshi Kurihara. 2018.Proposition of Autonomous Interactive Desk Environment. In Proceedingsof NGHAI’2018. Southampton, UK, December 2018, 3 pages.

1 INTRODUCTIONRecent advances in science and technology have led to progress inhigh performance and lower price of computer hardware and sen-sor devices, making it easy to harvest, store and analyze real-timebehavior information of people in real space. With the progress ofscience and technology like this, active researches about the net-work of various devices has been conducted. That is called Internet

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).NGHAI’2018, December 2018, Southampton, UK© 2018 Copyright held by the owner/author(s).ACM ISBN 123-4567-24-567/08/06.

of Things (IoT), and owing to the spread of IoT, the "UbiquitousComputing Infrastructures" have been prepared[3]. This meansthat everyone can use various information and services anywhereat any time. On the infrastructure, however, users need to select theinformation and services and customize them according to theirneeds, therefore "Ambient Computing Infrastructures" was newlyproposed. On the novel infrastructures, the ambience surroundsusers and autonomously interacts with them[4].

With many expectations for using the ambient computing in-frastructure, a lot of studies have been undertaken toward practicalapplication in a number of fields. Since agents in those ambientsystems are environments, some possible situations to use them arehouse, office and so on. Actually, some of the related works aimedat the improvement of comfort in those situations by controllingair conditioning, lighting, back ground music, etc[2][1][8]. On theother hand, we believe that it is also important to improve thecomfort of working environments (closer to hand) in homes andoffices. Therefore, in this study, we propose an interactive desk-top work support system named Autonomous Interactive DesktopEnvironment (AIDE).

2 AIDE: AUTONOMOUS INTERACTIVE DESKENVIRONMENT

AIDE is a system that supports people who perform desktop work.Specifically, a task that an operator is about to perform is inter-actively recognized, and windows and real objects themselves au-tonomously move to an appropriate position. The desktop workrefers to working with personal computers, such as writing docu-ments with word processors, developing a software, writing some-thing with a pen on a printed document. Figure 1 shows an appear-ance of AIDE. It consists of three components: a notebook PC, arobot arm for moving desk items and a web camera for observingdesk items. The camera is mounted on the robot arm to specify theposition of objects on the desk. Examples of AIDE’s operation areas follows.

• When a worker brings a document to the desk, AIDE recog-nizes the task of the worker, and a staple as an object agentmoves to the operator’s hand using the robot arm. And thestapler removes itself by the robot arm as well, observingthe end of the operator’s work.

NGHAI’2018, December 2018, Southampton, UK E. Takaya et al.

Figure 1: An appearance of AIDE

• On the display there are editors for programming and win-dows of browsers as agents. When a worker newly launcheda word-processing software, AIDE judges that the task ofthe worker has changed from programming with viewing abrowser to document creation with viewing a browser. Andthe windows of the browser and word-processing softwarechange to proper position and size which are easy for theoperator to use.

In order to support such works as described above, AIDE has thefollowing functions.

• Function to recognize the task that an operator is about toperform.

• Function for object agents andwindow agents to autonomouslymove to appropriate positions.

By implementing these functions, AIDE can recognize what kind oftask the worker will do in the future, based on names of applicationwindows on the display and types of real objects on the desk. Andthe associated window agents and object agents can autonomouslymove to the appropriate position for the task.

In the following, we describe the definitions of the object agentand the window agent in this study, and the methods for the taskrecognition of workers.

2.1 Object AgentsReal objects used in desk work such as pens, staplers, cups, etc. areregarded as object agents. Each agent is required to autonomouslymove to an appropriate position according to the a task of a worker.And a robot arm is used for these movements. Each object agenthas a QR code to record the name, coordinate, orientation and timestamp of the object using web camera. The record is taken everysecond. In this paper, however, the function of the object agent issupplementary.

2.2 Window AgentsApplication windows displayed on a PC monitor is regarded aswindow agents. Each of them is required to move to an appropriate

position on the display according to a task of an operator. As withobject agents, title, coordinate of each side, and order of windowscounted from the forefront are recorded every second.

2.3 Recognition of Worker’s Task andMovement of Agent

The AIDE system interactively recognizes tasks of workers andperforms appropriate actions. In order to make this possible, in-formation of each agent recorded every second is represented asa matrix. Task clustering and recognition are performed with thematrix.

2.3.1 Generation ofWorkMatrix. Logs of agents described aboveare combined to generate a matrix. We call it a work matrix in ourstudy. An example of a work matrix is shown in figure 1, and its el-ements represent the usage rate of agents per minute. For example,if an agent is used for 30 seconds in a minute, the correspondingelement is 0.5. Furthermore, the window agents in this work matrixis weighted by TF-IDF algorithm[6]. This process is performed sothat window agents commonly used in various tasks but not active(foremost) have relatively low values, while those with long activetime are relatively high values. As a result, the main window agentin each task has a large value, which facilitates clustering.

2.3.2 Task Clustering. Let X (its size is I× J) be the work matrixweighted by TF-IDF, and it is decomposed into factor matrices T(its size is I×K) and V (its size is K× J) by non-negative matrix fac-torization (NMF)[5]. Then, agent clusters used in each task appearsin the factor matrix V. The number of clusters K, that needs to bespecified when NMF is calculated, is determined using copheneticcorrelation coefficient (CPCC)[7] which is an evaluation index ofclustering. For details on NMF and CPCC, see [5] and [7].

2.3.3 Movement of Agents. The cluster, obtained by the methoddescribed above, is used for the task recognition when the sametask is performed next time. When a task is recognized, each agentmoves according to the position associated with the cluster of thetask.

2.4 Exploratory ExperimentIn this study, two preliminary experiments were conducted and eval-uated. One was to verify the task recognition accuracy of workersand the other was to validate the action of AIDE. In each experi-ment, there were three men (A, B and C) as workers in common.They were accustomed to using a PC. The devices used for the ex-periments are a laptop (windows 7 enterprise), a robot arm (WidowX Robot Arm Kit Mark II), and a web camera with a wide anglelens.

2.4.1 Evaluation of Task Recognition Accuracy. Three workerswere asked to perform two to four different tasks, and the accuracyof clustering by AIDE was calculated. Tasks performed by eachworker are shown in Table 1.

The accuracy of clustering for work by three people were verifiedwith rate of matching between the number of clusters and thenumber of actually performed tasks. And the F-measure showedwhether the agents included in each cluster are appropriate . Theresults are shown in Table 2. Although recognized clusters in C’s

Proposition of Autonomous Interactive Desk Environment NGHAI’2018, December 2018, Southampton, UK

Table 1: Tasks performed by each worker

Worker task1 task2 task3 task4A Create Briefing Paper Coding Create an E-mail NAB Translation Schedule Coordination Create an E-mail Create an ArticleC Create an Article Coding NA NA

Table 2: The accuracy of task recognition

Worker TCA F-measuretask1 task2 task3 task4

A 1.0 (3/3) 0.84 0.86 0.86 NAB 1.0 (4/4) 1.0 0.84 1.0 0.84C 0.5 (2/4) 1.0 0.75 0 0

Table 3: Evaluation of AIDE’s behavior

Worker task1 task2 task3 task4A 3 4 4 NAB 2 4 3 4C 3 1 NA NA

work were more than actual, it could be said that a certain degreeof accuracy is achieved.

2.4.2 Evaluation of Human-Agent Interaction. In addition to theabove experiments, we verified the behavior of AIDE based on taskrecognition. Task clusters of each worker are those obtained by theabove experiment. Evaluation was carried out with a questionnaireof 1 (bad) to 4 (good) per task. The results are shown in Table 3.Although it was only a preliminary experiment and has few samples,it could be said that comfort of AIDE was shown to some extent.

3 CONCLUSIONSIn this paper, we proposed AIDE, which is an ambient systemfor people who work on desks. In our system, desktop objects andapplicationwindows are regarded as agents, and they autonomouslymove appropriately according to the task of a worker. User’s worklogs are represented by a matrix, and clustering is performed byNMF to recognize the worker’s tasks. Each agent moves accordingto the recognition. Preliminary experimental results showed therecognition accuracy of the task and the validity of the interactionto some extent. But as a future work, it is necessary to constructa system that is more suitable for the reality. Specifically, we areplanning to add the following functions.

• To localize the position of objects without the QR codes.• To observe the situation of workers.

In any case, deep learning technology, which has been remarkabledevelopment in recent years, will be useful.

REFERENCES[1] Diane J Cook, Juan C Augusto, and Vikramaditya R Jakkula. 2009. Ambient

intelligence: Technologies, applications, and opportunities. Pervasive and MobileComputing 5, 4 (2009), 277–298.

[2] Michael Friedewald, Olivier Da Costa, Yves Punie, Petteri Alahuhta, and SirkkaHeinonen. 2005. Perspectives of ambient intelligence in the home environment.Telematics and informatics 22, 3 (2005), 221–238.

[3] Jayavardhana Gubbi, Rajkumar Buyya, Slaven Marusic, and MarimuthuPalaniswami. 2013. Internet of Things (IoT): A vision, architectural elements,and future directions. Future generation computer systems 29, 7 (2013), 1645–1660.

[4] Brad Johanson, Armando Fox, and Terry Winograd. 2002. The interactiveworkspaces project: Experiences with ubiquitous computing rooms. IEEE pervasivecomputing 1, 2 (2002), 67–74.

[5] Daniel D Lee and H Sebastian Seung. 2001. Algorithms for non-negative matrixfactorization. In Advances in neural information processing systems. 556–562.

[6] Gerard Salton and Michael J McGill. 1986. Introduction to modern informationretrieval. (1986).

[7] Robert R Sokal and F James Rohlf. 1962. The comparison of dendrograms byobjective methods. Taxon (1962), 33–40.

[8] Miguel A Zamora-Izquierdo, José Santa, and Antonio F Gómez-Skarmeta. 2010. Anintegral and networked home automation solution for indoor ambient intelligence.IEEE Pervasive Computing 9, 4 (2010), 66–77.