See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/327424414

IoT Big Data Analytics for Smart Homes with Fog and Cloud Computing

Article  in  Future Generation Computer Systems · September 2018

DOI: 10.1016/j.future.2018.08.040

CITATIONS

3READS

1,445

4 authors:

Some of the authors of this publication are also working on these related projects:

Smart Meters Big Data View project

A Vision System for Date Fruit Harvesting Robot View project

Abdulsalam Yassine

Lakehead University Thunder Bay Campus

72 PUBLICATIONS   628 CITATIONS   

SEE PROFILE

Shailendra Singh

Lakehead University Thunder Bay Campus

9 PUBLICATIONS   89 CITATIONS   

SEE PROFILE

M. Shamim Hossain

King Saud University

212 PUBLICATIONS   2,505 CITATIONS   

SEE PROFILE

Ghulam Muhammad

King Saud University

233 PUBLICATIONS   2,173 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Abdulsalam Yassine on 04 September 2018.

The user has requested enhancement of the downloaded file.

IoT Big Data Analytics for Smart Homes with Fog and

Cloud Computing

Abdulsalam Yassinea, Shailendra Singhb, M. Shamim Hossainc, GhulamMuhammadd

aDepartment of Software Engineering, Lakehead University955 Oliver Road, Thunder Bay, Ontario, P7B 5E1, Canada

(Email: ayassine@lakeheadu.ca)bDepartment of Electrical and Computer Engineering, Lakehead University

955 Oliver Road, Thunder Bay, Ontario, P7B 5E1, Canada(Email: ssingh59@lakeheadu.ca)

cDepartment of Software Engineering, College of Computer and Information Sciences,King Saud University, Riyadh 11543, Saudi Arabia, (E-mail: mshossain@ksu.edu.sa)

dDepartment of Computer Engineering, College of Computer and Information Sciences,King Saud University, Riyadh 11543, Saudi Arabia (Email: ghulam@ksu.edu.sa)

Abstract

Internet of Things (IoT) analytics is an essential mean to derive knowledgeand support applications for smart homes. Connected appliances and devicesinside the smart home produce a significant amount of data about consumersand how they go about their daily activities. IoT analytics can aid in per-sonalizing applications that benefit both homeowners and the ever growingindustries that need to tap into consumers profiles. This article presents anew platform that enables innovative analytics on IoT captured data fromsmart homes. We propose the use of fog nodes and cloud system to allowdata-driven services and address the challenges of complexities and resourcedemands for online and offline data processing, storage, and classificationanalysis. We discuss in this paper the requirements and the design compo-nents of the system. To validate the platform and present meaningful results,we present a case study using a dataset acquired from real smart home inVancouver, Canada. The results of the experiments show clearly the benefitand practicality of the proposed platform.

Keywords: Internet of Things (IoT), Cloud Computing, Fog Computing,Big Data Analytics, Energy Management, Smart Homes

Preprint submitted to Elsevier July 9, 2018

1. Introduction

Smart homes are the theme of the future living. Many communities acrossthe globe are currently deploying smart homes as part of modernization ini-tiatives. These always-on houses generate massive amount of valuable datafrom smart devices and appliances connected to an IoT system [1]. Theability to analyze these data in near real-time and off-line allows for thediscovery of various information that has significant impact on our society’ssafety, health, and economy. For example, a smart city’s health care sys-tem can determine the status of patients inside a smart home by monitoringtheir usage of appliances and detect their routine or abnormal activities thatcould indicate signs of health problems [2][3]. A utility company may analyzelarge amount of energy consumption data from appliances inside the hometo learn about the behavior of occupants and recommend electricity bill re-duction plans for consumers based on energy usage profiles [4]. Such scenarioleads to cost reduction not only to consumers but also to utility companies.Real-time IoT application allows manufactures to analyze data continuouslyand determine or predict an appliance maintenance schedule or promptly re-place malfunction equipment. These examples of IoT applications reveal theadvantages of analyzing smart home data. While such data presents valuableopportunities in understanding the dynamics and behavior of smart homesand their occupants, it also spells out a tremendous challenge regarding datamanagement, storage, and analytics. To ensure that users are not drowningin floods of data, they need systems capable of managing, analyzing andtransforming this amount of data into actionable insights for smart city ap-plications that demand prompt actions with stringent requirements. Thesesystems must also meet the needs of scalability with the growing volume ofdata and the temporal granularity of decision-making whether it is off-lineor near real-time.

In this paper, we propose a system which combines IoT and big dataanalytics technology with fog and cloud computing. The proposed systemaddresses the challenge of designing efficient solutions that are fast and canhandle large volumes of unstructured smart home data. In this system,fog computing provides fast near real-time analytics while the abundance ofcomputing and storage resources in the cloud system is used to carry outcomputationally intensive applications. The process involves taking data-in-motion from IoT sources, such as individual smart meters, appliances, anddevices and integrate them for highly sophisticated analytics processes that

2

deliver timely decision-making. Fog computing nodes are resource-efficientbecause they are equipped with virtual machine technologies capable of con-tinuously processing fresh IoT streams of data and transfer the processeddata to the cloud for further processing [5]. Cloud computing offers a multi-tude of benefits such as Infrastructure as a Service (IaaS): providing access tounlimited storage space, Platform as a Service (PaaS): potential to executeresource-intensive applications, Software as a Service (SaaS): facilitates soft-ware access, and Utility Services: store massive volume of data for remoteaccess. Fog computing play a critical role in the IoT ecosystem to supportthe processing of big data for near real-time responses. Furthermore, fogcomputing fundamentally processes and stores data at the edge of the cloudsystem [6]. This unified architecture allows us to resolve the latency issuespertaining to the underlying transport communication network of cloud sys-tems which has a significant impact on time-sensitive applications [1][7] [8][9].

For the evaluation of the proposed system, we present a case study ofanalyzing and processing streams of data from a smart home. The smarthome generates continuous streams of the massive data in short time inter-vals. Processing and analyzing such data is vital for many applications (e.g.,healthcare systems, smart grid energy management applications etc.) [10][11]. The main contribution made in this paper are as follows:

• Proposing a platform for IoT smart home big data analytics with fogand cloud computing. The system design allows the processing of mas-sive multiple smart home IoT data in distributed fog nodes, which ac-commodate cognitive data mining algorithms that provide insight fromprocessed data. This approach is rather significant for many applica-tions that require access to information for timely functional economiesof scale, where smart home operations can be cost-effectively deployedand used.

• Providing detail requirements and design component analysis of theplatform architecture. Specifically, we discuss requirements of scalabil-ity regarding the processing of multiple IoT data streams from smarthomes and the design aspects of minimizing communication overheadbetween the fog nodes and the cloud systems. Furthermore, we dis-cussed the design of mechanisms for task allocation, IoT managementand integration services, and admission authentication of smart homeand third-party applications.

3

• Presenting a case study of an actual smart home. We analyzed thesmart home IoT data for behavioral and predictive analytics of occu-pants pertaining to energy consumption routines and patterns. We dis-cussed the applications of these finding within the context of demandresponse management and electricity cost reduction. These analysisare considered among the primary functions and applications of smarthomes, which can be scaled with fog and cloud computing to an entiresmart community[[34]].

The organization of the paper is as follows: in the next section, we discussthe related work. In section 3, we present the components of the proposedplatform followed by a study case in section 4. Finally, in section 5 we presentthe conclusion of the study and provide direction for future work.

2. Related Work

Recently, several studies have proposed systems and frameworks for IoTdata analytics using various architectures involving fog and cloud computing.In this section, we discuss these studies especially those that are representa-tive of the state-of-the-art and close to our work.

Many researchers tackled issues closely related to our work such as thosein [14] [33][29][24][31] and [25]. For example the work in [14], focuses on pre-dictive analytics for smart homes that need access to historical data whichmust be stored in a large database that can only be provided by a cloudsystem. The work in [33] investigated the smart home services for in-depthanalysis of home appliance frequent pattern usage. Specifically, the discoveryof co-utilization behavior of appliances inside smart homes. For this purposethe authors propose a multidimensional patterns mining framework from alarge number of residential users connected to an Internet Service Provider(ISP). The authors in [29] developed a new gateway system to automaticallyintegrate and configure new home-based IoT devices for seamless analyticsin cloud systems. The SLASH framework in [24] presents new approach forsmart home adaptivity and self-learning mechanisms. The idea include thedevelopment of big data layer with an analytical engine that supports occu-pants behavior. The work in [25] proposes an end-to-end home automationsystem that supports multiple IoT protocols for data acquisition and analy-sis. The authors claim that their system is capable of handling data comingfrom city-wide deployed devices. Similar to the work in [25], a general smart

4

city paradigm is proposed in [31] for IoT big data analytics system that in-tegrate sensors from smart homes, traffic, vehicles, surveillance sensors, etc.using Hadoop ecosystem real-life environment.

In addition to the above mentioned studies, the authors in [40] presentreal-time data analytics engine that facilitates processing of data near thesource of information. The proposed analytical engine ensures that data isprocessed before it is offloaded to the central cloud system. The systemcoordinates the analytics between the physical location of the IoT devicesin the vicinity leading to the creation of device-to-device analytical layerunder the cloud system. The main issue with this approach is that it addscomplexities to the system to the point that makes it practically prohibitive.Similar to [40], the works in [15][18] and [19] address the issues of dataanalytics at the edge of the cloud system, but focused on the latency problemof processing large amount of IoT data using fog computing. The designapproach in these systems brings resources of edge computing as close aspossible to the source where data is generated. The work in [36] furtherinvestigates this issue and develops mechanisms to estimate the latency forcloud-fog-IoT continuum systems.

For real-time analytics of IoT data in uncontrolled environments, the workpresented in [32] proposed a general-purpose IoT framework that integratewireless hub nodes to support analytical reliability and assures real-time dataacquisition. The work in [35] proposed a system that runs data analytics ina distributed fashion using fog computing, IoT devices, edge and centralservers. The main approach is to optimize the decision-making of analyticssuch that all IoT devices are fairly treated and satisfied. The results of thiswork show a promising solution for enhancing the utilization of fog and cloudcomputing systems. To facilitate intelligence of the edge network in providingrobust analytics for IoT systems, the work in [37] outlined a new approach todynamically automate the transitions between the central cloud system andits edge taking into account the various conditions and requirements of theapplications. The author in [26] proposes a general model and architecturethat ingests IoT data streams into fog computing nodes. The model addressesthe challenges of existing techniques and the shortcomings pertaining to theessential dimensions of data analytics related to system, data, human andoptimization.

It is important to note that a platform with fog computing nodes coupledwith cloud computing offers a resource-efficient processing of IoT big dataat near realtime basis while providing insights and processed data to cloud

5

for further processing and analysis. This integrate design facilitates us toaddress the latency issues of cloud system that can have a remarkable impacton time-sensitive applications.

Our work in this paper is in line with the work presented in [25], however,our focus is on a scalable IoT big data analytics platform with fog computingthat is capable of managing, analyzing and transforming household energyconsumption data into actionable insights. Therefore, we present a holisticarchitecture that is suitable for an end-to-end analytics of IoT connectedsmart homes. We discuss the validity of the architecture and the intercon-nectivity of analytical modules. For the evaluation of the system model, wepresent a case study of data streams collected from an actual smart homein Vancouver, Canada. Our case study addresses the challenges of data an-alytics of smart home energy consumption for smart grid applications (e.g.Automatic Demand Response). It must be noted that this work differs from[4][3][27] and [28]. These works do not address the IoT big data analyticsin fog and cloud computing systems, but focus on analyzing behavioral en-ergy consumption that lead to peak hours as in [4], activity recognition forhealthcare applications [3], and prediction models [28]. This paper intro-duces detail system requirement and component design analysis for an IoTbig data analytics platform for smart homes via fog and cloud computing.The smart home dataset, the platform as well as the results in this paper arecompletely different from our previous work.

3. Platform Overview

The fast deployment of smart homes is taking off across the world, andit is becoming a compelling business opportunity for various industrial ap-plications. Smart homes that are supported by IoT paradigm generate largeuseful data. However, unlocking the potential of this information hinges onthe development of sophisticated big data analytics tools and platforms capa-ble of processing, analyzing and managing these data in cost-effective ways.In this section, we address the system requirements for the development ofIoT big data with fog and cloud computing, and we present the componentsof the proposed platform.

3.1. Platform Requirements

The design of innovative platforms that are suited to support a largeamount of data generated from smart homes posses peculiar requirements,

6

functionalities, and design structures.

• Resource Distribution: Processing large amount of data generatedfrom household appliances and devices require cost-effective and re-source efficient big data analytics closer to the physical system. Miningcontinuous streams of IoT data should meet the timing requirements formany smart city applications such as automatic demand response, sen-sitive healthcare applications, safety and surveillance operations, etc.which require predictable latency for near real-time detection and noti-fication. Mostly, these functionalities face serious constraints when pro-cessing data and invoking services from the back-end cloud. The prox-imity of resources helps overcome the high-latency that is associatedwith the provisioning of cloud-based services. Therefore, optimizedscheduling mechanisms are required to coordinate the tasks among fogcomputing nodes and should appropriately allocate resources from thecloud system.

• Scalable Analytics: Data streams from smart homes present a chal-lenging prospect for processing and mining operations. These datastreams are received at the system in high volumes, high velocity andfrom various sources. Furthermore, smart home data change over timedue to the changing behavior of occupants. Hence, such data requirescalable tools to process and analyze them for different behavioraltraits. Also, these data come from different communication channelsthat contribute to the abnormalities and noises that require furthercleaning and pre-processing to reduce their dimensionality and betterextract useful information.

• Performance: IoT data streams should be handled in a parallel man-ner to boost the performance of data analytics and to optimize thesmart home operations. Depending on the analytics activity, the spe-cific requirements include elastic resource acquisition, efficient data net-work management, data reliability, and functional data abstractions.Furthermore, IoT data processing should make full use of all compu-tational resources to address performance challenges of near real-timeand off-line computation algorithms such as finding concealed patterns,quickly and valuable knowledge. Furthermore, there is a need for co-ordination between fog computing nodes and the cloud system. Fognodes cannot carry out high computational tasks because of resource

7

and storage limitations, and therefore, intensive-applications are per-formed in the cloud servers for better performance [30].

• Integration: At any single time instant, data collected from differentdevices inside smart homes are unstructured and cannot be processedusing conventional tools. Therefore, IoT data from smart homes re-quire dedicated integration mechanisms to be further processed in aunified system. Such approach imposes restrictions on edge computingselection concerning add-hoc pre-processing operations. As a result, anautomated and adaptive data ingestion methods must be developed tointegrate incoming varying data rates and volumes and accommodatevarious data sources and applications requirements. These methodsshould process all inbound time-series data, execute data transforma-tions, and coordinate the transmission of the processed data to thecloud system.

• Visualization:, Another major requirement for dealing with large-scale IoT data from smart homes, is data-relevance through visualiza-tion. This requirement is rather significant since different applicationsrely on the visualization of the underlying trends and the perceivedpresentation of data after being processed and analyzed for decision-making. Application-specific data visualization serves the domain ex-pertise by providing meaningful results based on data quality. Forexample, presenting abstraction of data clustering for co-utilization ofappliances inside a smart home leads to better understanding the be-havior of occupants inside the house and their activities.

To satisfy the above requirements, in this paper we present an IoT bigdata analytics platform for processing and analyzing a large volume of smarthomes data streams. Next subsection describes the proposed platform.

3.2. Platform Design Components

Figure (1) shows the architecture of the proposed platform. It consistsof IoT big data analytics with fog computing nodes and cloud system. Thecomponents of the platform support complex operation of continuous in-tegration, processing and analytics of multiple smart home data. The fognodes broaden the services of the cloud system to the network edge closeto the physical locations of the smart homes, thus allowing for faster dataprocessing and services applications that can only be served within specific

8

time constraints. The cloud system takes the heavy lifting of processingcomputationally intensive application.

In the proposed model, smart homes are the source of data. Such datatypically arrive at the smart home IoT gateway from different sources in-cluding household appliances and smart devices. The acquisition of datais typically performed by specific IoT protocols such as machine-to-machine(M2M)/Message Queuing Telemetry Transport (MQTT) that communicatewith smart home devices and IoT gateways. An IoT gateway acts as anagent that mediates between the smart home and the cloud system. The IoTgateway may also provide local processing and storage functions including au-tonomously controlling and filtering of data streams. In the proposed model,the IoT gateway can be used to serve multiple households while ensuringtrusted connectivity and security by enforcing policy-based access mecha-nisms. The acquisition of data during this communication process passesthrough several stages until the data rest on cloud storage devices wherefurther processing may be performed in future. As mentioned in section ”In-troduction”, smart home data have the volume, velocity, and variety char-acteristics to be considered as big data. The analytics operations includefiltering and cleaning, clustering and aggregation where each operation takesextensive time depending on the nature of the data. The following are thedetails of the platform components.

• Smart Home Components: The smart home consists of sensors, de-vices, appliances and metering systems. The components of the smarthome are roughly categorized into three tiers namely cyber-physical,connectivity, and context-aware. The cyber-physical tier consists ofsmart devices, metering systems, sensor systems, appliances, electri-cal vehicle charging points, and energy management systems. Theseelements are responsible for all actual operations of the smart homeand are installed on the home premises. They interact with the out-side world through the connectivity tier. As mentioned earlier in thissection several communication protocols allow these devices to com-municate among themselves and with the cloud system through an IoTgateway. The connectivity tier is responsible for outbound and inboundcommunication with the smart home, which includes direct interactionwith the occupants through mobile or web applications. The context-aware tier provides real intelligence about the essence of the smarthome. Context-aware tier includes user-defined rules and policies to

9

Figure 1: IoT Big Data Analytics with Fog Computing

10

manage the interaction and the services of the smart home. This tierallows for context-based privacy and security configuration that sat-isfies the occupants’ concerns. Activity recognition, event detection,behavioral and predictive analytics are performed by the fog and cloudcomputing system and reported to the smart home applications. Forexample, behavioral analytics can be very effective to understand howusers go about using their appliances and derive conclusions about en-ergy consumption, which can be used to forecast the future demand.Activity recognition can be used to allow caregivers in healthcare appli-cations [3] to detect abnormal behavior of patients. The applicabilitiesand benefit of such analytics and services are countless.

• IoT Management and Integration Services: The IoT manage-ment service is a broker-based subsystem responsible for handling IoTservice requests from multiple smart home applications into the cloudsystem. It plays a vital role in providing authentication services forthese requests and ensures that the rules of admission are consistentwith the pre-configured policies in the rules engine. Services that re-quire access to data about the smart home must register with the IoTmanagement broker before using any data from the cloud system. Theorchestration of these tasks are handled by the requests handler afterregistration and authentication. The operation of the IoT managementservices are protocol independent and are responsible for maintainingcontinuity and flexibility for the whole IoT ecosystem. Figure (2) illus-trates the operation of the IoT management services. In this figure allservices are first authenticated and then registered before accessing thedata. Figure (3) shows a high level description of the integration ser-vices in the proposed platform. This service provides seamless accessto external applications via programming interfaces (APIs). The mainapproach in this design is the decoupling of the analytical componentsof the fog nodes and the cloud system from user applications. By so do-ing, the integration service assures security since external applicationswould not be able to have any direct access to the analytical engine.Also, it adds data abstraction by enabling the use of data for varioususer-specific applications including mobile and desktop. Another mainadvantage is providing interoperability for various channel technologies(Wi-Fi, Bluetooth, ZigBee, and LPWAN etc.)and data transfer proto-cols M2M, MQTT, CoAP (Constrained Application Protocol), AMQP

11

Figure 2: The IoT Big Data management service is responsible for request handling,authentication and service registration

Figure 3: The IoT Big Data integration service is responsible for smart home functionaland third party services and applications

(Advanced Message Queuing Protocol)), Websocket etc..

• Fog Computing Nodes: The fog node provides additional resourcesand computational services to support various smart home time-sensitiveapplications. The fog nodes provide the means for accelerating ana-lytics services while ensuring increased responsiveness from the cloudinfrastructure. Figure (1) shows the main functions of the fog nodein our platform. It is composed of several functions including datapre-processing, pattern mining, classification, prediction, and visual-ization. These functions are responsible for rapid analytics of smarthome data collected through the IoT system while the aggregated re-sults are sent to the cloud or directly to the serviced applications. Thefog node performs all the short-term analytics at the edge of the cloud

12

system. Data arrived at the fog node are unstructured and do not havea predefined model. During the cleaning/pre-processing process, er-rors, redundancies, and outliers are removed to ensure consistency. Inthe pre-processing stage, all IoT streams are filtered, parsed and trans-lated into a unified data structure for further analysis. At this stage,raw data which contains millions of high time-resolution data recordsare transformed into a pre-defined resolution for each device. The fre-quent pattern mining techniques are conducted on the data to discoverthe occurrence of appliance correlation in data streams. Frequent pat-tern mining searches for these recurring patterns in a given dataset todetermine associations and correlations among patterns of interest[39].In clustering stage, we employ an unsupervised form of classificationwhich is capable of distinguishing classes of appliances learned fromthe data [39]. Prediction analytics are responsible for forecasting oc-cupants activities or use of certain devices. Visualization provides aninteractive medium for the user to discover knowledge from data toenhance the decision-making process. Finally, the results of the stagesmentioned above are sent to the cloud system which has an abundanceof resources for computationally intensive tasks.

It should be noted that the configuration of the fog node to ensure theprivacy of the smart home is a challenging prospect. As fog nodes arebecoming a major computation hub, smart home private data becomevulnerable to various attacks. Therefore, a new breed of trust manage-ment systems and privacy protection mechanisms are required to tacklesuch problem. These mechanisms are not considered in this paper.However, other remedies for this problem can be found in [20][21][22]and [23].

• Cloud System: In the proposed platform, the cloud system is respon-sible for providing core services to smart home applications that includehistorical data analytics, extended storage capabilities, and core smarthome management infrastructure. The cloud services include smarthome device tracking, configuration, analysis, reporting, authentica-tion and authorization services. These functions provide value-addedservice for users to control and manage their smart homes using differ-ent means ( e.g., web and mobile applications) as well as to interactwith third-party vendors. Also, the cloud system provides heavy com-putational using large-scale data mining resources such as MapReduce,

13

Spark, Storm, etc. The cloud system uses its back-end computation togain business insight and updated the fog nodes about new operationalrules.

4. Case Study

In this case study, we perform IoT data analytics of appliances anddevices from a smart home in Vancouver, British Columbia, Canada.The data is publicly available from Harvard Education website [41].The dataset consists of one minute interval measurements of multiplesmart home appliances over the span of two years, April 2012-April2014. We perform data analytics on IoT streams to uncover occupantsbehavior of appliance usage such as identifying frequent patterns asso-ciated with appliances including hour of day, day of week, and monthof year as a means of understanding how occupants go about their dailyroutines. For this particular study, we assume that the data is acquiredat the fog node where an analytical engine is responsible of perform-ing immediate analysis to satisfy the requirements of applications suchas energy consumption management, targeted advertisement, activityrecognition. Ideally, for this particular case study a scalable computingresources are required to enhance the performance with additional ac-quisition of data. Our tests are conducted on a single node consistingof computer system running i5-Core CPU with 8GB of RAM and 1 TBof storage device. The main processing resources are allocated to theanalytics part where we process 2-years of data. The current runningtime takes few minutes which can be improved with more computingresources, however, it shows that one fog node is capable of processingmore than one smart home.

In this paper, we focus on energy consumption analytics of IoT datafrom smart homes. Specifically, we analyze IoT data aggregate of en-ergy measurements of home appliances. An application such as residen-tial Automatic Demand Response (ADR) requires energy consumptiondata about appliances in residential homes to be analyzed to engagethem in demand response signals effectively [12]. To realize this con-cept, we first assume that every home is attached to a fog node that isresponsible to perform the analytics. The aim is to analyze the frequentpattern of home appliance usages, the context of appliance usage (i.e

14

co-utilization of appliances), the cluster of appliances with respect tothe time of use (i.e spatio-temproal analysis), and the forecast of appli-ance usage. The following steps illustrate the life-cycle of the analyticsat the fog node.

Data Cleaning and Preparation: The dataset contains millions ofrecords (sample of raw data is shown Table (1)) with a large amountof data about appliances. Data about appliances is collected everyminute for a length of two years (April 2012-April 2014). These datameasurements include: unix timestamp, line voltage, voltage, apparentpower. The process of cleaning the data started by importing the datafiles in Python scripts. The cleaning process includes eliminate un-necessary columns, convert Unix timestamp to human readable date,remove values that are below the standby power threshold, removingoutliers and duplicate rows. The entire cleaning process was completedusing Python with regular expressions (RegEx). The preparation ofthe data includes comparing all the reading to a pattern and only thematching patterns were stored in a database. The tuples not matchingthe pattern are considered noise because the values for power and times-tamp are supposed to be Integers only, hence, any different characterin this values would represent an error in the recording process of thosetuples. The process of pattern matching also ensures the quality of thedata, because any tuple that was incomplete or inconsistent did notmatch the pattern and therefore was ignored. For the purpose of train-ing, we developed a synthetic dataset which include the appliance, thetime of its operation, the date, and the power. With this informationin hand, we can then perform clustering analysis and frequent patternof appliance usage such as the hour of day, day of week, month of theyear.

Frequent Pattern Mining: For frequent pattern mining, we are in-terested in analyzing the occurrences of when certain appliances arebeing used by examining the ”ON/OFF” state and the energy con-sumption. Being in an On state allows for the inference that a humanis currently using a particular appliance. This information can be ben-eficial in certain applications, and as a result, the data and patternsmined have a value to industries. For example, by knowing when anindividual is likely to have the television turned on could help compa-nies target advertisements. We would like to derive these patterns in

15

Table 1: Sample of IoT Data from the Smart Home

Timestamp* Apparent power consumption of appliances**1360548360 210- -1360548420 70- -1360548480 28* - Unix timestamp** - Example of appliances includes Dishwasher, Toaster, TV, DryerHome Theater, Washing Machine, Laptop etc.

discrete-valued relations [28]. Specifically, we study the appliance us-age patterns of the whole house and seek to uncover associations overtime domains. Formally, let A be a database consisting of n itemsetsT1 such that A= (T1, T2, ...Tn). An itemset is considered a frequentpattern if it appears with a certain frequency in a database transac-tion. The user may define the threshold level of the frequency countof an itemset in a transaction. One of the methods of determining thefrequency count is known as the support count s which is defined asthe statistical count of the frequency of an itemset in a transactioncarried over the database A. For example, two itemsets I (I ⊆ A)and J (J ⊆ A) are counted as frequent patterns in a transaction iftheir support sI and sJ is above a threshold value known as the mini-mum support minsup. In the case of finding a frequent pattern, thenthe association rules are determined. An association rule is expressedas {I ⇒ J} and are derived from the support − confidence, wheresupport sI⇒J such that s(I ⇒ J) = sI⇒J = s(I ∪ J) is the percentageof all transactions that have (I ∪ J) in A. The support represents themutual preconditions of this association in the database while the con-fidence is the preconditions that contribute to the consequence. In thissense, the frequency of itemsets in a transaction suggests the statisticalsignificance of the association rule ( meaning the probability P (I, J)),

determined by the confidence |s(I∪J)||s(I)| (meaning the conditional prob-

ability P (I|J)) [38][39]. We employ the frequent pattern FP-Growthalgorithm [38][39] and its extension [4] on this smart home dataset.Procedure (1) shows the steps of capturing the frequent patterns from

16

Figure 4: Hour of the Day Energy Consumption Pattern - (a) Dishwasher (DWE), (b)Cloth Dryer (DWE), (c) Kettle (KE), (d) TV, (e) Home Theater (HTE), (f) Laptop (EBE)

the dataset. Figures (4), (5), and (6) show the pattern of energy con-sumption of six appliances in the home comprising hours of day, day ofweek, month of year. We applied a minimum support threshold of 30%on the dataset and turned all values that were below the threshold to 0and all the ones above to 1. This allowed us to obtain a binary matrixto check what appliances are in use at the specific time as shown in thetable (2).

The final result of the frequent pattern mining is the association amongappliances that are the result of the simultaneous use of the applianceby occupants. Figure (7) shows an example of hourly use and day ofthe week use of appliances. From figure (7-a) it is apparent the twoappliances used the most together are the dishwasher and television be-tween the hours of 6pm-1030pm. For the three appliances (dishwasher,dryer, and television) on at the same time, the most likely time of theday this will happen between 8-8: 30 pm. The days of the week infigure (7-b) demonstrates that very often the dishwasher and televisionare frequently on together at the same time. Inspecting each day in-dividually, you can see certain patterns such as Monday and Tuesdaynight the dishwasher and television are on the longest amount of timeor Saturdays the television and dishwasher are on later at night.

17

Procedure-1:Generating Frequent Patterns-GP Growth

Require: Start by constructing the Frequent Pattern Tree(FP-T)with algorithm in [28].

Ensure: Generation of Frequent Patterns1: Start by single path in the FP-Tree2: Check for the support s among all the combination of

nodes in the FP-Tree3: Determine the frequent patterns4: Add to Database A the discovered patterns5: Repeat steps for multiple paths in FP-Tree6: Determine the frequent patterns using for multiple paths7: Add to Database A the discovered patterns8: Final Frequent Patterns = Single path ∪ Multiple path9: Determine association rules based in %30 support using

algorithm in [4]10: Create a binary matrix and update it with all appliance

patterns within the threshold.

Figure 5: Day of the Week Energy Consumption Pattern - (a) Dishwasher (DWE), (b)Cloth Dryer (DWE), (c) Kettle (KE), (d) TV, (e) Home Theater (HTE), (f) Laptop (EBE)

Cluster Mining: The above frequent pattern analysis provided insight

18

Table 2: Sample of a binary matrix to uncover the frequent pattern of appliance usages

10:30pm-11pm

11pm-11:30pm

11:30pm-12pm

DWE-CDE

DWE-TVE

CDE-TVE

DWE-CDE-TVE

0 0 0 1 1 1 10 0 0 1 1 1 10 0 0 0 0 0 00 0 0 0 0 0 00 0 0 1 1 1 10 0 0 1 1 1 11 0 0 1 1 1 10 0 0 1 1 1 1

Figure 6: Day of the Week Energy Consumption Pattern - (a) Dishwasher (DWE), (b)Cloth Dryer (DWE), (c) Kettle (KE), (d) TV, (e) Home Theater (HTE), (f) Laptop (EBE)

about how the smart home occupants are co-utilizing their appliances. Clus-tering analysis allows us to interpret time-intervals associated with groupsof appliances. This is rather important to uncover deeper behavior of ap-pliance energy consumption of specific times (e.g. peak hours). To achievethis objective, we implement the k-mean clustering algorithm in [39]. Thebasic principle of the k-mean algorithm is that it defines k centers whichare placed in specific positions away from each other. Then, the functionG(z) =

∑ki=1

∑Cij=1(||ai−bj ||)2 is used to determine the squared error value,

where ai − bj is the Euclidean distance between a and b, Ci represents thenumber of data points in ith cluster. Determining the optimal number of k is

19

Figure 7: Frequent Pattern of appliances- Dishwasher (DWE), Cloth Dryer (DWE), TV -(a) Hour of the Day (b) Day of Week [D1 -Sunday - D5 Saturday ]

vital for getting better results. There are many methods for determining theideal number k as described in [43]. The approach in this work is using thesilhouette coefficient as a means of calculating the optimal number k [44].This method basically measures the quality of the cluster by evaluating howwell the data points are positioned within a cluster. It computes the averagedistance of yj given as xj = average{dis(yj , yi)} to all other data points incluster Ci and then determine wj = min(wj) across all the clusters except

Ci. The Silhouette coefficient for yj is determined as ryj =(wj−xj)

max(xj ,wj)

and the Silhouette coefficient for cluster Ci and for having k clusters asrCi = average(syj ) for j = d1..dn and rk = average(sCi) for i = 1..krespectively. The higher the average silhouette value, the better the cluster-ing. In other words, the average Silhouette provides observation about thevarious values of k ∈ 1, 2, 3...m, where m represents the unique objects ina dataset. To find out the optimal number of clusters, the process is con-tinuously executed and the average Silhouette coefficient is calculated untilfinding the optimal number of clusters k that maximizes rk.

Figure (8) shows the clustering of appliances at the hour of the day, wherecluster strength signifies the frequency of use of appliance, i.e., a higherstrength of a cluster for an appliance indicates the higher use of it duringthe period. Higher or lower usage of appliance, i.e., patterns of applianceusage can be the direct representative of energy consumption behavior ofoccupants. Such an analysis can be conducted at various levels such asindividual house, group of houses, community or neighborhood, or at thesystem level. When done at a higher level such as neighborhood or systemlevel, the outcomes can help profile houses according to energy consumptionbehavior and customize demand response mechanism to be more efficient.Further, at a single home, the outcomes can assist adapt recommendationsto reduce household energy cost while respecting the occupants expected

20

Figure 8: Smart Home Appliance Clustering - Hour of the Day

comfort. Moreover, it feasible to consider renewable energy generation atthe neighborhood or house level to fine-tune demand response programs orenergy reduction recommendations.

5. Conclusion and Future Work

In this paper, we presented a platform for smart home IoT big data ana-lytics with fog and cloud computing. We provided detail requirement analysisand illustration of the platform components. The process of performing theanalytics in the fog node is presented, and the results show the possible ap-plications of the system in different aspects. For example, applications of thedata acquired may include activity recognition to identify health problems,identify energy consumption patterns and energy saving planning, and pre-dict appliance maintenance schedule to avoid expensive repairs and ensureefficient operations from the point of view of energy consumption.

In general, the platform can aid effective and in-time decision making forindividual house owners by facilitating various energy management programsat home level. Household energy consumption management and data ana-lytics is a complex operation that requires continuous integration of multiplesources into a common processing system with easy access to data. Otherpossible application may be extended to serve companies who are interestedin targeted advertisement. They can choose a time slot where customersare using these appliances; typically between 6 pm and 9 pm because the

21

residents are frequently watching television at this time. Analytics in fognodes increases the ability of the platform to manage an integrated array ofIoT data streams for various applications in highly automated ways whichresult in significant savings for service providers. Also, service providers candesign and develop their applications using fog nodes that offer abundanceelasticity to enhance performance, redundancy and storage devices for theirapplications.

For future work, we plan to develop optimization mechanisms such asthose in [16][17] to determine the optimal distribution and configuration offog nodes while taking into consideration the computational resources and ca-pability of processing the required data from multiple homes. Furthermore,we plan to refine the platform component and test with different datasetsfrom various homes. This approach is crucial to validate the applicability ofthe platform and its robustness in dealing with all kind of IoT data measure-ments. We also plan to study a benchmarking scheme to assess and capturethe performance of the platform and analytics under different concerns in-cluding runtime, CPU utilization, data size, incoming requests, etc.

6. References

[1] H. El-Sayed et al. Edge of Things: The Big Picture on the Integrationof Edge, IoT and the Cloud in a Distributed Computing Environment.IEEE Access, vol. 6, pp. 1706-1717, 2018

[2] P. Rashidi, D. J. Cook, L. B. Holder and M. Schmitter-Edgecombe.Discovering Activities to Recognize and Track in a Smart Environment.IEEE Transactions on Knowledge and Data Engineering, vol. 23, no. 4,pp. 527-539, April 2011

[3] A. Yassine, S. Singh and A. Alamri. Mining Human Activity PatternsFrom Smart Home Big Data for Health Care Applications. IEEE Access,vol. 5, pp. 13131-13141, 2017. doi: 10.1109/ACCESS.2017.2719921

[4] S. Singh and A. Yassine. Mining Energy Consumption Behavior Patternsfor Households in Smart Grid. IEEE Transactions on Emerging Topicsin Computing. doi: 10.1109/TETC.2017.2692098

[5] M. Marjani et al. Big IoT Data Analytics: Architecture, Opportunities,and Open Research Challenges. IEEE Access, vol. 5, pp. 5247-5261, 2017

22

[6] A. Mebrek, L. Merghem-Boulahia and M. Esseghir. Efficient green solu-tion for a balanced energy consumption and delay in the IoT-Fog-Cloudcomputing. IEEE 16th International Symposium on Network Computingand Applications (NCA), Cambridge, MA, 2017, pp. 1-4, 2017

[7] A. S. Chhabra, T. Choudhury, A. V. Srivastava and A. Aggarwal. Predic-tion for big data and IoT in 2017. International Conference on InfocomTechnologies and Unmanned Systems (Trends and Future Directions)(ICTUS) Dubai, 2017, pp. 181-187

[8] Y. Ge, X. Liang, Y. C. Zhou, Z. Pan, G. T. Zhao and Y. L. Zheng.Adaptive Analytic Service for Real-Time Internet of Things Applica-tions. IEEE International Conference on Web Services (ICWS), SanFrancisco, CA, 2016, pp. 484-491

[9] P. Pouladzadeh, S.V.B. Peddi, P. Kuhad, A. Yassine, S. Shirmoham-madi. A virtualization mechanism for real-time multimedia-assisted mo-bile food recognition application in cloud computing, Cluster Comput-ing. 18 (3) (2015) 10991110

[10] A. R. Al-Ali, I. A. Zualkernan, M. Rashid, R. Gupta and M. Alikarar. Asmart home energy management system using IoT and big data analyticsapproach. IEEE Transactions on Consumer Electronics, vol. 63, no. 4,pp. 426-434, November 2017

[11] A. Berouine, F. Lachhab, Y. N. Malek, M. Bakhouya and R. Ouladsine.A smart metering platform using big data and IoT technologies. 3rdInternational Conference of Cloud Computing Technologies and Appli-cations (CloudTech), Rabat, 2017, pp. 1-6.

[12] A. Yassine. Implementation challenges of automatic demand response forhouseholds in smart grids. 3rd International Conference on RenewableEnergies for Developing Countries (REDEC), Zouk Mosbeh, 2016, pp.1-6

[13] M. Frincu, C. Chelmis, M. U. Noor, and V. K. Prasanna. Accurate andefficient selection of the best consumption prediction method in smartgrids. Proc. IEEE Int. Conf. Big Data, 2014, pp. 721729

23

[14] H. Cai, B. Xu, L. Jiang and A. V. Vasilakos. IoT-Based Big Data Stor-age Systems in Cloud Computing: Perspectives and Challenges. IEEEInternet of Things Journal, vol. 4, no. 1, pp. 75-87, Feb. 2017.

[15] J. He, J. Wei, K. Chen, Z. Tang, Y. Zhou and Y. Zhang. Multi-tier Fog Computing with Large-scale IoT Data Analytics for SmartCities. IEEE Internet of Things Journal, vol. PP, no. 99, pp. 1-1. doi:10.1109/JIOT.2017.2724845

[16] M. Taneja and A. Davy. Resource aware placement of IoT applicationmodules in Fog-Cloud Computing Paradigm. IFIP/IEEE Symposiumon Integrated Network and Service Management (IM), Lisbon, 2017, pp.1222-1228

[17] Q. T. Minh, D. T. Nguyen, A. Van Le, H. D. Nguyen and A. Truong.Toward service placement on Fog computing landscape. 4th NAFOSTEDConference on Information and Computer Science, Hanoi, 2017, pp. 291-296

[18] N. M. Gonzalez et al. Fog computing: Data analytics and cloud dis-tributed processing on the network edges. 35th International Conferenceof the Chilean Computer Science Society (SCCC), Valparaiso, 2016, pp.1-9.

[19] H. Cao, M. Wachowicz and S. Cha. Developing an edge computing plat-form for real-time descriptive analytics. IEEE International Conferenceon Big Data (Big Data), Boston, MA, 2017, pp. 4546-4554.

[20] A. Yassine, A. A. Nazari Shirehjini and S. Shirmohammadi. Smart Me-ters Big Data: Game Theoretic Model for Fair Data Sharing in Dereg-ulated Smart Grids. IEEE Access, vol. 3, no. , pp. 2743-2754, 2015.

[21] A.Yassine, S.Shirmohammadi. Measuring user’s privacy payoff using in-telligent agents. Computational Intelligence for Measurement Systemsand Applications, 2009, CIMSA’09. IEEE International Conference on,pp169-174

[22] A. Paverd, A. Martin, and I. Brown. Security and Privacy in Smart GridDemand Response Systems. Series Lecture Notes in Computer Science.Volume 8448, pp 1-15, Smart Grid Security, Springer, 2014

24

[23] A.Yassine, S.Shirmohammadi. Privacy and the market for private data:a negotiation model to capitalize on private data. IEEE/ACS Interna-tional Conference on Computer Systems and Applications, Doha, 2008,pp. 669-678.

[24] M. Sultan and K. N. Ahmed. SLASH: Self-learning and adaptive smarthome framework by integrating IoT with big data analytics. ComputingConference, London, 2017, pp. 530-538.

[25] J. Lohokare, R. Dani, A. Rajurkar and A. Apte. An IoT ecosystem forthe implementation of scalable wireless home automation systems atsmart city level. TENCON 2017 IEEE Region 10 Conference, Penang,2017, pp. 1503-1508.

[26] S. Yang. IoT Stream Processing and Analytics in the Fog. IEEE Com-munication Magazine, vol. 55, no. 8, pp. 21-27, 2017.

[27] S. Singh, A. Yassine. IoT Big Data Analytics with Fog Computing forHousehold Energy Management in Smart Grids. SGIoT 2018 - 2nd EAIInternational Conference on Smart Grid and Internet of Things, NiagaraFalls, Canada, 2018

[28] S. Singh, A. Yassine. Big Data Mining of Energy Time Series for Behav-ioral Analytics and Energy Consumption Forecasting. Energies, 2018,11, 452

[29] B. Kang, D. Kim and H. Choo. Internet of Everything: A Large-ScaleAutonomic IoT Gateway. IEEE Transactions on Multi-Scale ComputingSystems vol. 3, no. 3, pp. 206-214, July-Sept. 1 2017.

[30] J. Y. Kim, H. J. Lee, J. Y. Son and J. H. Park. Smart home web ofobjects-based IoT management model and methods for home data min-ing 17th Asia-Pacific Network Operations and Management Symposium(APNOMS), Busan, 2015, pp. 327-331

[31] M. M. Rathore, A. Ahmad and A. Paul. IoT-based smart city develop-ment using big data analytical approach. IEEE International Conferenceon Automatica (ICA-ACCA), Curico, 2016, pp. 1-8

25

[32] G. Daneels et al. Real-Time data dissemination and analytics platformfor challenging IoT environments. Global Information Infrastructure andNetworking Symposium (GIIS), St. Pierre, 2017, pp. 23-30.

[33] G. Poghosyan, I. Pefkianakis, P. Le Guyadec and V. Christophides. Ex-tracting usage patterns of home IoT devices. IEEE Symposium on Com-puters and Communications (ISCC), Heraklion, 2017, pp. 1318-1324

[34] J. He, J. Wei, K. Chen, Z. Tang, Y. Zhou and Y. Zhang. MultitierFog Computing With Large-Scale IoT Data Analytics for Smart Cities.IEEE Internet of Things Journal, vol. 5, no. 2, pp. 677-686, April 2018

[35] H. J. Hong, P. H. Tsai, A. C. Cheng, M. Y. S. Uddin, N. Venkatasub-ramanian and C. H. Hsu. Supporting Internet-of-Things Analytics ina Fog Computing Platform. IEEE International Conference on CloudComputing Technology and Science (CloudCom) Hong Kong, 2017, pp.138-145

[36] J. Li, T. Zhang, J. Jin, Y. Yang, D. Yuan and L. Gao. Latency estima-tion for fog-based internet of things. 27th International Telecommunica-tion Networks and Applications Conference (ITNAC), Melbourne, VIC,2017, pp. 1-6.

[37] P. Patel, M. Intizar Ali and A. Sheth. On Using the Intelligent Edgefor IoT Analytics. IEEE Intelligent Systems vol. 32, no. 5, pp. 64-69,September/October 2017.

[38] J. Han, J. Pei, and Y. Yin. Mining frequent patterns without candidategeneration. 2000 ACM SIGMOD International Conference on Manage-ment of Data USA, pages. 1 12, 2000

[39] J. Han, J. Pei, Y. Yin, and R. Mao. Mining frequent patterns withoutcandidate generation: A frequent-pattern tree approach. Data Miningand Knowledge Discovery vol. 8, no. 1, pp. 5387, 2004

[40] F. Mehdipour, B. Javadi and A. Mahanti. FOG-Engine: Towards BigData Analytics in the Fog IEEE 14th Intl Conf on Dependable, Auto-nomic and Secure Computing. 14th Intl Conf on Pervasive Intelligenceand Computing, Auckland, 2016, pp. 640-646

26

[41] S. Makonin, B. Ellert, I. V. Bajic, F. Popowich. AMPds2 - Almanac ofMinutely Power dataset: Electricity, water, and natural gas consump-tion of a residential house in Canada from 2012 to 2014. Scientific Data,DOI 10.1038/sdata.2016.37, vol. 3, pp. 1-12, 2015.

[42] T. Kanungo, D. M. Mount, N. S. Netanyahu, C. D. Piatko, R. Silvermanand A. Y. Wu. An efficient k-means clustering algorithm: analysis andimplementation IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 24, no. 7, pp. 881-892, Jul 2002.

[43] C.A. Sugar and G.M. James. Finding the Number of Clusters in a DataSet: An Information Theoretic Approach. J. Am. Statistical Assoc., vol.98, pp. 750-763, 2003

[44] P.J. Rousseeuw. Silhouettes: A graphical aid to the interpretation andvalidation of cluster analysis. Journal 702 of Computational and AppliedMathematics 1987, 20, 5365.

27

View publication statsView publication stats