Service platform for context reasoning in home environment

Mikko Sallinen, Topi Pulkkinen Industrial Machine-to-Machine

Systems VTT Technical Research Centre of

Finland Oulu, Finland

Mikko.Sallinen@vtt.fi

Young-Sung Son, Jun-Hee Park Green Conputing Department

Electronics and Telecommununications Research

Institute (ETRI) Daejeon, Republic of Korea

ysson@etri.re.kr

Yann-Hang Lee School of Computing, Informatics, and Decision Systems Engineering

Arizona State University (ASU) Tempe, Arizona, USA

yhlee@asu.edu

Abstract—In this paper, we introduce a service platform for context reasoning to be used in various home applications. The main functional idea of the platform is to collect and combine data from individual services e.g. Google weather and location and make intelligent decisions based on semantic modeling and ontologies. Two example applications presented here are smart home heating and ubiquitous home care for a diabetes patient.

Keywords-component; energy management; ontologies;

I. INTRODUCTION Management of home appliances, networks and sensors

have been intensive research topic in several years, starting from vision of Internet of things [1], going through RFID and it’s applications and ending up to latest applications on are such as Smart Grid [2, 3, 8]. One important part of enabling technologies are Wireless Sensor Networks [4, 5] which are used widely in many applications starting from industry [6] to healthcare at home [7, 13].

One key issue of the intelligent context aware service platform is the interoperability. It means different components of the systems including devices and services are logically and semantically defined so data relationships can be maintained and applications can share the information. One common way to avoid interoperability problem is to use open framework such as OSGi (Open Service Gateway Initiative) which helps communication between the services and store the data into an ontology model. This approach has been realized e.g. in [9]. There are also common ontologies such as SOUPA which are proposed to be used in the ubiquitous and pervasive applications [10, 11]. The result of the work was a generic framework that could easily adopt different applications that will work together.

In our work, we have also chosen OSGi to be the method for interfacing services. In this paper, we introduce a platform which will integrate sensors and cloud services with a decision making machine to manage different applications in home environment. Basically, this approach accommodates information from various sources, which enables us to develop applications for different domains easily. The applications that are introduced in this paper are smart home heating and ubiquitous healthcare (uHealth) for diabetes patient.

This paper is organized as follows: chapter 2 gives an overview to the platform, chapter 3 introduces the ontology model for the context information, chapter 4 describes the implementation of the platform and chapter 5 draws the conclusions.

II. DESCRIPTION OF THE CONTEXT REASONING PLATFORM The service platform has capability to utilize versatile

information in different applications. This has been achieved by a multi-layer structure, which gathers the information from the appliances, stores the raw data, abstracts it, and utilizes ontology reasoning to provide additional data or to control the services. The four layers that perform these tasks are “Home network middleware layer”, “Abstraction layer”, “Context layer” and Service layer”, which are presented in Fig. 1.

A. Home Network Middleware layer There are service frameworks to gather and manage home

service information like DPWS (Device Profile for Web Services) and OSGi (Open Service Gateway initiative). Some appliances utilize existing home network protocols such as UPnP, KNX, LonWorks etc. and finally there are legacy appliances with serial interface or no networking interface at all. For the platform presented in this paperi we adopted OSGi and utilized HeRA (Home Resource Architecture) [14] as an OSGi bundle to interface home devices easily. OSGi provides a Java based service registry where each connected module publishes its interface.

B. Abstraction layer To abstract the raw data we utilized software agents that

can store the actual raw data into database and also to process the time series data into a manageable form such as user-centric values. For example it is possible to utilize static information on the context layer such as user’s home location and calculate user’s current distance to home based on dynamic raw data for example phone GPS. This information could then be inserted in a different instance in the ontology.

The agents collect data from the home network middleware layer and executing some processing functions. The platform supports centralized ontology based reasoning and therefore the agents don’t perform any complex BDI (Belief-Desire-Intention) based planning or communication

978-1-4673-4828-7/12/$31.00 ©20122 IEEE ICTC 2012

between other agents. One important part of abstraction layer is also to be able to visualize information from the raw data. The abstraction layer also holds the functionality for pattern finding and matching based on the queries from the Context layer.

C. Context layer Purpose of the context management layer is to maintain

ontologies for specific domains but in the same time consider relationships between the domain ontologies. Operations of this layer support RDF formatted data management such as adding instances, inserting data values to data- and object properties and delivering data to the service layer. Context layer can also execute pattern matching functions in abstraction layer based on the ontology structure. This enables the platform to find new user or environment related data and their relationships. Such function could be utilized when trying to find user’s behavior pattern from history data or some behavior models in different domain. For example, pulse (heartbeat) information can be used in both home and uHealth domain.

D. Service layer The service layer executes integrated services based on the

reasoning results from the Inference Module (see Fig. 5). It can request information from ontology instances at context layer and further aggregate data if required. An example of data aggregation would be defining the level of warm (cold, cool, warm, hot) based on the temperature reading from a thermometer, user’s preference and additional weather condition (rainy, windy, snowy, sunny). This information can

then be utilized in heating application that controls the operation of the heaters.

III. INFORMATION MODEL OF THE PLATFORM Bettini et al. [12] considered the targets of context

modeling as follows (i) the easiness to digitalize the real-world concepts (ii) the effectiveness of context information model, (iii) the support for reasoning, and (iv) the easiness and scalability of context information management.

In this regard we designed our framework’s information model as a service centric interoperability problem. An individual service publishes some functions to produce new data or to handle a specific task e.g. turning a device on/off. The platform can invoke these functions during and after the decision making process by creating jobs which are passed to the services. To be able to utilize the various data sources in different application domain rules, the data has to be abstracted to the correct level of detail which is presented in the following ontology model.

According to modeling, by dividing the concept hierarchies in to domains and mapping these domains using the upper domain ontology is a good practical method for data aggregation. Our proposed ontology consists of upper domain ontologies namely “service” and “user”, which contain very generic concepts for linking underlying domain ontologies such as “home device domain” and “uHealth domain”. The general outline of the ontology structure is described in Fig. 2. It also describes the concepts that are closely related user’s life pattern

Figure 1. Service platform operational layers to support context based reasoning.

302

Figure 2. Ontology structure where domain ontologies are mapped together

using general concepts found in service and user ontology.

information such as “temporal” and “location”. The

directional arrows illustrate object properties between the main ontology classes.

The detailed ontology model for uHealth domain is presented in Fig. 3. The HealthMonitor class contains three different monitor sub-classes HealthEventAbstractedValue, HealthEventCrispValue and HealthEventTemporalStatus. These monitors are utilized based on the target information: for example the instance HealthEventTemporalStatus describes a long-term temporal state of health e.g. fever with start and end times, while the HealthEventCrispValue can describe a medical variable such as SMBG (Self Monitoring Blood Glucose). For pre-processed data that needs to be abstracted the HealthEventAbstractedValue can be utilized. Abstraction of the information is required if some health related incident needs to be measured with several medical sensors and afterwards the data is fused such as insulin shock.

In Fig 3. The user class has a relationship with the HealthStatus class, which in turn has relationship with the Symptom class and the Disease classes. These classes are meant to be extended to cover the mapping to UMLS (Unified Medical Language System).

IV. IMPLEMENTATION OF INTELLIGENT SERVICE PLATFORM

Proposed platform was tested in an intelligent home environment to provide automatic and assisting tools and services for the user. One target of these tools was to optimize the usage of the energy (electricity consumption at home) in case of home heating. Another target is to visualize and recommend an activity to a user based on the behavior model and goals. The behavior monitors concentrated on exercise, eating and medical condition of the user. In both cases, we aimed to use most common and practical data sources such as

Figure 3. uHealth domain ontology classes (bottom) with Service and User

mapping classes.

as mobile phone, web calendar, weather services and location services such as GPS. In the case of monitoring the state at home, we included additional sensors e.g. noise, temperature and humidity to get more diverse data about the living conditions.

A. Smart energy management at home To test the system, a following set-up was constructed: a

user has one heater at home and a smartphone to use. Person goes to work regularly and wants to save the energy consumption at home by controlling the heater in optimal way. Availability of information as well as the services providing the required data is described in Table I:

TABLE I ENVIRONMENT DATA USED IN DEMONSTRATION, CASE ENERGY MANAGEMENT AT HOME

Information Service

Location of the user Mobile phone GPS

Weather Internet weather service

Traffic Dummy data from mobile phone

User’s schedule Internet calendar application

Home temperature measurement Home device interoperability system

Heater operation (on/off) Home device interoperability system

The system should provide the expected temperature when

the user arrives at home. During the time when user is absent the energy consumption should be kept minimal. The scenario is as follows:

� The user will leave the workplace at some time T and the reasoning engine will inference

1) when the user will arrive at home

303

2) when to turn on the heating device

3) how to react to the context information changes.

In addition to the dynamic data, the system already has some static information available like user’s temperature preferences, the average energy consumption of the heater, efficiency of the heater, BIM (Building Information Model) of the house and simplified models how different weather affects the traffic speed and need for the heating. The demonstration utilizes OWL-DL semantic model to handle the data relationships and SWRL rules to create axioms from the data model.

1) Software architecture

The high-level software architecture is presented in Fig. 5 where the bottom shows the Inference Module (IM) and the top the plug-in services. These are connected together with a service framework which hosts services. The data is saved in an OWL format and the application rules are created with SWRL. The implementation utilizes a Jess rule-engine which invokes the service agents based on the inference results. When an external event is triggered, the service agents can place the event data into the ontology and invoke the inference module if necessary.

When we compare the platform layers presented in Fig. 1 and software architecture in Fig. 5, is apparent that service framework in Fig. 5 is operating on both Home Network layer and on Abstraction layer. In Home Network layer the service framework interfaces the services published by the devices or 3rd party services such as internet calendar, while on the Abstraction layer the service framework processes the information so it is suitable for the OWL format. It is important not to confuse the interfaced services that are connected to the service framework with the Service layer that combines simple services together to produce more advanced services such as “Home Energy Management” service or “Diabetes Home Care” service.

In the ontology, all services can register their underlying functions, so the applications, which are using the individual services, can easily invoke correct service. As an example, let’s assume that a person has a thermometer at home whereas another person has an air condition with a temperature sensor. Both devices provide the same functionality “temperature measurement” which can be used by other applications. This service enables the application developer to design an application that is usable in different environments. Of course this means the service agents and their functions need to be registered to the model. This process is illustrated in Fig. 4 sequence diagram. In the beginning the information model (OWL-file) is loaded by the operator to the inference module and then the related agents can register themselves. After registration, each agent will activate the reasoning procedures related to the services it can provide. When a registered service is identified and activated as a part of inference results, a “workticket” is generated that provides information to access the service. The worktickets are then retrieved by the different agents. When new data arrives to the model, the agents have permission to run reasoning (DoInference in the sequence diagram, Fig 4) to activate the rule engine which derives new

Fig.4. Registering the service and application agents into OWL model.

inference results. Because many parts of an application require data processing that is outside the capability of SWRL, the software agents can perform pre- or post-processing. An example would be calculating heating requirement based on the BIM (Building Information Model), measured temperature, weather and preference temperature, which is difficult to express with rule language. In the case that the agents notice no change on data values, they will not activate the rule engine.

2) Testing of the system

The demonstration uses some existing software modules from a Java service framework to manage plug-in services, and a device interoperability system, which can communicate with the different home networks. Also we utilized an existing Java based ontology modeling API for ontology management and a rule-based reasoning engine for context based reasoning. Firstly, the real world concepts are captured by combining a cloud service and several individual home service agents that transfer the cloud service data to the context information model.

Also here we utilized Android mobile phone, generic internet services, and home device interoperability system. In the demonstration scenario the Android application first sends its GPS location and dummy traffic data to a TCP server via JSON interchange format and the TCP server then provides an OSGi interface for the software agents to request data from the server. The identification to the server is done with a mobile phone number.

Android platform is also running a “result receiver manager” module which is used to show a simple timestamp when the user’s heater will be turned on after the inference module has received the necessary data and calculated the time.

For the calendar and weather data we developed a simple Java application for interfacing Google weather and Google calendar internet services and abstracted the data for the demonstration. Basically we were interested the case when the user arrives to home and there is a need for heating/cooling operations according to weather information and user’s calendar data. The final information source was device interoperability system called Home Resource Architecture (HeRA) [14].

304

Figure 5 Software architecture of the demonstration test case.

Instead of a real test environment, we utilized a simulated temperature sensor and a virtual UPnP heating device whose state was controlled by HeRA. The heating device was a simple on/off device and the temperature sensor reading could be changed manually by web interface provided by HeRA. HeRA was also included in OSGi service framework as a plug-in service.

The test results show that the performance of our system is suitable for online control of the home devices as the delays from starting inference cycle to execution are in between 1-5 seconds, which is enough for most services.

B. Monitoring of the diabetes patient at home Second test case of the system consists of the measuring

and decision making tools and technologies for the person that lives at home and has diabetes. The long-term goal of the system is to monitor the behavior and habits of a diabetes patient and find out any health changes during a long period of time. The short-term target is to find if there is a possibility to predict up-coming dramatic change in the health state of the patient e.g. a dangerous situation caused by incorrect action by the patient. Accident cases will also be monitored and reported.

Active monitoring system for diabetes consists of sensor devices, medical devices, intelligent appliances and a smart phone. For understanding the user activities, the system should fuse the sensing data inside and outside of the home.

For precise monitoring of diabetes blood glucose meter is the most important medical sensor of the system. The other sensors monitor user’s activities indirectly and based on the knowledge base reason the user’s status. The collected sensing data would create a fundamental database for understanding user’s life patterns that are related to the disease. Outside the home, the system can gather data of the user’s activities with sensors of the smart phone: accelerometer, gyroscope and GPS. Also user’s input can be utilized via an online questionnaire.

Fig. 6. Diabetes active monitoring system.

Questionnaire is also helpful when verifying or classifying the measured data. All gathered data should be stored at Log Storage, which the system can use for analyzing and visualizing the user’s patterns and progress. To evaluate this system, we have implemented two scenarios:

1) One person has a regular life pattern but the system should monitor that enough exercises will be carried out to manage the diabetes.

2) Another person lives busy and unstable working life even if the doctor has instructed several things about lifestyle including medicine, sleeping, eating and exercise. In the test scenario, an insulin shock will occur at home and the system should recognize it and alarm help to the person.

In the scenario 1 we use person’s online calendar to study his daily pattern. Different tasks are reserved in different format in his calendar: work, hobby, shopping, exercise etc. In addition to this, other information about person’s physical status such as weight and blood pressure will be combined

305

together to form the overall figure of the health status. If there is not enough exercise, system proposes additional exercise times and locations via the calendar and reminds the user with mobile phone. The system will also give guidance about eating and drinking by comparing the actual status to the optimal case. Scenario 2 follows the first scenario by monitoring the person’s status. Difference to first case, there is also knowledge about serious insulin shock, how to avoid them and how to behave if such occurs.

Table 2 presents the basic information sources for monitoring user’s activity. Firstly, the system estimates user’s outdoor activity with his smart phone. It records all the movement and activity of the day with a context information client. It will be used when validating user’s current situation with knowledge logic description. Whenever some conditions are matching, smart phone can trigger additional questionnaire with co-related GUI. Within the house, a camera will detect the user’s activity. It uses depth images which detects the general body movement, but not the person. This enables us to estimate the user’s pose at home without harming the privacy of the person. The privacy is also maintained by only gathering abstracted information such as user’s states (lying, standing, sitting) and the user’s location and not the raw data.

TABLE II . ENVIRONMENT DATA USED IN DEMONSTRATION, CASE

MONITORING THE PERSON AT HOME Information Service Technique

Location of the user (indoor)

WLAN / camera Image Motion Detection algorithm

Location of the user (outdoor)

Smart phone with GPS GPS Spatial-Temporal analysis

Activity of the user Smart phone with sensors, noise sensor, camera

Signal analysis, Image Motion Detection algorithm

Weight of the user Body scale

User’s schedule Internet calendar app

Task / activity of the user

Sensors in home appliances: refrigerator, microwave, questionnaire

Temporal data analysis & mining

Weather Google weather service

Personal symptoms Questionnaire

Personal health information

Doctor’s prescription Knowledge logic description

Secondly, some environment sensors such as humidity and

CO2 will record long term living conditions at home. Also body scale and blood glucose meter are important devices for monitoring the user’s actual medical status. Through a long term record, the system can analyze the progress of the diabetes and find some relationships with related living patterns.

V. CONCLUSIONS In this paper we presented intelligent service platform that

has capability to collect data from different sources and make decisions and control appliances at home based on that information. One core idea and main contribution of this work is capability of the system to exchange the information between

the different applications by using the ontology mapping. The decision making is based on modeling the information using ontology models and JESS engine to make decisions. The overall system is implemented using OSGi service framework which enables connectivity to the other systems easier.

We implemented our system in two different applications: energy management at home and monitoring of the diabetes patient at home. Both cases illustrated that they require careful design of the ontology to enable efficient services and exchange of data. However, it is beneficial to collect data from several applications and fuse it to achieve new information.

ACKNOWLEDGMENT This work was conducted using the Protégé resource, which

is supported by grant LM007885 from the United States National Library of Medicine. Authors want to thank Korea Institute of Advancement of Technology (KIAT) and VTT from the funding of this project.

REFERENCES [1] ITU Report, 2005: The Internet of Things. www.itu.int/internetofthings/ [2] Michahelles F., Thiesse F.; Schmidt A.; Williams J., Pervasive RFID

and Near Field Communication Technology. Pervasive Computing, IEEE, Volume 6, Issue 3, July-Sept. 2007, p. 94 – 96.

[3] Y. Son, T. Pulkkinen, K. Moon, and C. Kim, “Home energy management system based on Power Line Communication,” IEEE Trans. Consumer Electron., vol. 56, no. 3, pp. 1380-1386, Aug. 2010.

[4] Akyildiz I., Su W., Sankarasubramaniam Y., Cayirci E.. “A Survey on Sensor Networks”. IEEE Communications Magazine, Aug. 2002. pp. 102-114.

[5] Marron P. J., Minder D. Embedded WiSeNts Research Roadmap. ISBN 3-8325-1424-4. Logos Verlag Berlin 2006. 171p.

[6] Willing A. “Recent and Emerging Topics in Wireless Industrial Communications: A Selection”. IEEE Transactions on Industrial Informatics, Vol. 4, No 2., May 2008. Pp. 102-124.

[7] Bisiani R., Merico D., Mileo A., Pinardi S. ”A Logical Approach to Home Healthcare with Intelligent Sensor-Network Support”. The Computer Journal Advance Access published May 14, 2009. 20p.

[8] J. Lee, D. Jung, Y. Kim, Y. Lee, and Y. Kim, “Smart Grid solutions, services, and business models focused on telco,” Network Operations and Management Symposium Workshops (NOMS Wksps), 2010 IEEE/IFIP, pp. 323-326, April 2010.

[9] Gu T., Pung H., Zhang D. “Toward an OSGi-Based Infrastructure for Context-Aware Applications”. Pervasive computing, Oct-Dec 2004. Pp. 66-74.

[10] H. Chen, T. Finin, A. Joshi, “The SOUPA Ontology for Pervasive Computing” Whitestein Series in Software Agent Technologies, Springer. 2005.

[11] Chen H., Perich F., Finin T., Joshi A. “SOUPA: Standard Ontology for Ubiquitous and Pervasive Applications”. Int. Conf. on Mobile and Ubiquitous Systems Networking and Services. Aug. 22, 2004. 10p.

[12] C. Bettini et al. “A survey of context modeling and reasoning techniques”, Pervasive Mobile Computing (2009), Doi: 10.10 I6/j. pmcj.2009 .06.002.

[13] P. Valiente-Rocha and A. Lozano-Tello, “Ontology and SWRL-Based Learning Model for Home Automation Controlling,” in J.C. Augusto et al. (Eds.):Advances in Intelligent and Soft Computing Vol. 72 – Ambient Intelligence and Future Trends-International Symposium on Ambient Intelligence (ISAmI 2010, AISC 72), pp. 79–86

[14] Son Y.-S., Pulkkinen T., Moon K.-D., Kim C. Home energy management system based on power line communication. IEEE Transactions on Consumer Electronics. Vol. 56 (2010) No: 3, 1380-1386

306