Machine Learning in the Internet of Things for Industry 4.0

Tomasz Szydlo, Joanna Sendorek, Robert Brzoza-Woch, Mateusz Windak

AGH University of Science and Technology,Department of Computer Science, Krakow, Poland

Abstract

Number of IoT devices is constantly increasing which results in greater complexity of computations and highdata velocity. One of the approach to process sensor data is dataflow programming. It enables the developmentof reactive software with short processing and rapid response times, especially when moved to the edge of thenetwork. This is especially important in systems that utilize online machine learning algorithms to analyze ongoingprocesses such as those observed in Industry 4.0. In this paper, we show that organization of such systemsdepends on the entire processing stack, from the hardware layer all the way to the software layer, as well as onthe required response times of the IoT system. We propose a flow processing stack for such systems along withthe organizational machine learning architectural patterns that enable the possibility to spread the learning andinferencing on the edge and the cloud. In the paper, we analyse what latency is introduced by communicationtechnologies used in the IoT for cloud connectivity and how they influence the response times of the system.Finally, we are providing recommendations which machine learning patterns should be used in the IoT systemsdepending on the application type.

Keywords: Internet of Things, Edge Computing, Machine Learning, Industrial IoT

1. Introduction

Due to the advances in electronics and computersystems, we are observing an increasingly large num-ber of embedded devices connected to the Internet.Handling hundreds of thousands of interactions be-tween such devices is a very challenging task and ap-propriate organization of data processing is necessary.There are several aspects that have to be dealt with inthis context, including how, where and what computa-tions should be performed.

IoT systems often assume the interaction of deviceswith cloud services which makes the Service OrientedArchitecture (SOA) a suitable approach to designingprocessing logic. In that concept devices expose theirfeatures like services, which are then consumed byother services deployed in the cloud [1, 2]. This con-cept has since evolved into the so-called Web Of Things[3]. Nevertheless, the large volume of streaming datagenerated by IoT devices can make that approach in-efficient. The solution to this problem is to make pro-cessing data-driven and responsive. Flow-based pro-gramming [4] is a paradigm that defines applicationsin terms of flows of processes which exchange data bypassing messages across predefined connections. Theexecution time of operations is dependent on their or-der in the processing flow. This concept can be adapted

for IoT, where things are sources of data streams tobe processed. There are several runtime environ-ments that implement this concept, such as CppFBP[4], NoFlo [5], WotKit [6] and NodeRED.

Efficient flow processing of streaming data in IoT isespecially important in the industry, currently undergo-ing revolutionary changes colloquially referred to as In-dustry 4.0 [7]. The name refers to the notion of a fourthindustrial revolution, where the focus shifts to deliver-ing innovative services and products. Previous stagesof the industrial revolution introduced mechanizationthrough the use of (1) steam engines, (2) electricity (fa-cilitating mass production), and (3) IT solutions. Thefourth step is to use digital product models developedaccording to customer requirements and manufacturedby Smart Factories [8]. Such factories will be capableof self-planning and self-adaptation using devices com-patible with the concept of the Internet of Things andCyber-Physical Systems [9].

For optimal production of goods and their subse-quent delivery, Industry 4.0 proposes the use of ma-chine learning mechanisms to study customer require-ments and ensure optimal use of industrial equipmentand to enforce predictive maintenance. Self-planningand self-adaptation processes require a collection ofknowledge from the factories, development of modelsand applying them to obtain predictions. In such us-

Preprint submitted to arXiv May 25, 2020

arX

iv:2

005.

1114

6v1

[cs

.NI]

22

May

202

0

age, short response times are necessary. For exam-ple, in factory automation or motion control responsetimes lower than 10ms are mandatory, while processautomation can be satisfied with 100ms response times.Processing flows which contain machine learning algo-rithms can be executed in the cloud, offering high pro-cessing capacity, a global view of the IoT system andeasy management; however, this also introduces addi-tional time delay as data has to be streamed from sen-sors to the cloud. Deploying such flows only in the fog,on the network edge, limits knowledge sharing becausesensor data is not exchanged between remote premisesbut improves system response times. Lack of infor-mation exchange between branches may result in thesituation that the system will not be able to recognizeevents that have already occurred elsewhere.

In the paper, we present the generalized concept ofa flow-based processing stack and how the underlyinginfrastructure may influence dataflow execution. Wepropose three architectural patterns for machine learn-ing algorithms that can be applied to data flows prior totheir deployment and execution on real hardware. As aresult, the computation is divided into a part located inthe cloud, allowing for knowledge dissemination, andan edge part which provides high responsiveness. Theproposed patterns provide a tradeoff between the accu-racy of machine learning models and prediction times.In the evaluation of the concept, we are focusing on on-line supervised learning methods. We argue that sys-tems based on flow-based programming for IoT shouldtake into account a holistic view of the system [10]which perceive the system as a whole - in particular,the interplay of conflicting objectives and configurationoptions across all subsystems i.e. underlying hardware,communication topology, operating systems and execu-tion environments. The scientific contribution of thepaper includes (i) a generalized concept of a flow-basedprocessing stack for IoT, (ii) machine learning architec-tural patterns for IoT, (iii) evaluation of the communi-cation technologies used in the IoT systems for cloudconnectivity, and (iv) recommendations which patternand communication technology is appropriate for spe-cific types of IoT systems.

The structure of the paper is as follows. Section 2presents the state of the art of machine learning algo-rithms for embedded devices. In section 3 dataflow exe-cution is discussed. Section 4 discusses ML for IoT andproposes delta patterns. Section 5 contains an evalua-tion of the proposed concepts while section 6 sums upthe paper.

2. State of the art

In large-scale computing, TensorFlow [11] is amongwell recognised neural network-based machine learn-ing systems for distributed computing platforms. Dueto multi-layer design, it runs on multiple hardware plat-forms. Its computing tasks are dispatched to many ker-nels running on many CPU and GPU cores. The liteversion of the TensorFlow is a set of tools to run neuralnetwork models on mobile and embedded devices butthe machine learning model has to be optimized to runefficiently on resource constrained devices. The firstgroup of optimization techniques is related to the neu-ral network accelerators. Lane et al. [12] present asoftware accelerator that enhances deep learning ex-ecution on heterogeneous hardware, including mobiledevices. An original approach to machine learning inresource-constrained embedded systems is presentedby Bang et al. [13] where authors describe a hard-ware accelerator for performing deep-learning taskswhile requiring very low power to operate. The simi-lar chips are designed by Intel and Google. The secondgroup of solutions is related to the methods of neuralnetwork parameters quantization in order to diminishthe amount of computation data storage and transfertime [14]. There are also works related to other ma-chine learning algorithms. Shamili et al. [15] proposethe utilization of Support Vector Machine (SVM) run-ning on networked mobile devices to detect malware.Implementation of algorithms related to machine learn-ing domain on extremely resource-constrained deviceshas also been described, e.g. in [16, 17].

The increased interest in systems operating on theedge of the network has resulted in the research onmachine learning solutions that combines processingon the network edge and in the cloud. The work of Liuet al. [18] describes an approach for image recogni-tion in which the process is split into two layers: localedge layer constructed with mobile devices and remoteserver (cloud) layer. In the edge, i.e. on a mobile de-vice, an acquired image is preprocessed and segmen-tation is performed. Then the image is classified on aremote server running pretrained convolutional neuralnetwork (CNN). A more general survey on employingnetworked mobile devices for edge computing is pre-sented in [19]. There is also an ongoing research aimedat using machine learning methods to optimize energyconsumed by the devices. Peralta et al. [20] describethe method for sending sensor data from devices to thecloud that filters values which could be predicted by themachine learning methods.

In Industry 4.0, due to internal processes such asmachine ageing, an important aspect is the need for dy-namic adaptation to new patterns in the data or when

2

the data is generated as a function of time. This meansthat the aforementioned methods of moving learnedmachine learning models to edge devices are insuffi-cient because over time they become obsolete. Mod-els must be constantly updated to match the currentnature of the process. A comprehensive survey on in-cremental online learning is presented in [21]. Yin etal. [22] present an application of incremental learn-ing for object recognition. In contrast, [23] describesan application of learned navigation system to controlan autonomous robot. These publications, however,do not concern resource-constrained or embedded sys-tems that operate at the network edge or in a fog en-vironment. In this paper, we propose an organizationalpattern for such machine learning algorithms on theedge and in the cloud and then discuss its usage takinginto account the hardware capabilities of IoT systemsand the available communication technologies.

3. Dataflow execution

The kind of systems discussed in the paper followsthe concept of fog computing which was first describedby Cisco [24, 25]. Fig. 1 depicts the architecture of suchsystems. One of the most important factors for process-ing data in the IoT, especially for industrial IoT, is thefast response time. Table 1 contains the required laten-cies for critical IoT applications [26]. The processingflow can be executed in the cloud, providing virtuallyunlimited computational resources (e.g. Amazon AWS,Microsoft Azure or Google Cloud) and network band-width (e.g. fibre links in Core Internet), but the datatransmission latency (e.g. via GSM) might be unac-ceptable for timely reception of results. On the otherhand, executing data flows in the fog layer providesresponsiveness but lacks a global view of data. Thismeans that, for example, in a factory, the knowledgeacquired during the processing of sensor data in oneof its branches is not transferred to another branch.Because of this, the system may not detect potentiallydangerous situations that have already occurred.

Table 1: Required latencies for critical IoT applications

Use case Latency (ms)

Factory automation 0.25 - 10

Motion control 1

Tactile Internet 1

Smart Grids 3-20

Intelligent Transport Systems 10 - 100

Process automation 50 - 100

Speech recognition 350

Face recognition 500

Fig. 2 depicts examples of flow execution wheresome elements are executed on different devices.

Embedded System and Sensors

Fog Layer

Core IPv6 Network

Data Center/Cloud

Local networks

Backbone networks links

Backhaul and access links

LAN, RS485, MODBUS, BLE

3G, 4G, LTE, WiFi

IP/MPLS, BGP

Edge datacenters, Linux based devices

Constraintresources devices, FreeRTOS

Virtual Machines, SDN

Efficient backbone routers and switches

Figure 1: Fog computing architecture

Fig. 2a depicts a situation where data streams fromsensors are sent to the cloud for processing. This pro-vides the possibility to collect data from thousands ofsensors and process it at a single site, providing a highquality of results. On the other hand, such process-ing introduces huge latencies. In Fig. 2b processingis performed locally on the edge, which ensures shortprocessing times, but the information is not shared be-tween edges (e.g. several factories located on differ-ent premises). This solution can be regarded as exe-cution in a private cloud. Finally, Fig. 2c shows a sit-uation where the processing logic is spread betweenthe fog and the cloud. Sensor data is preprocessed inthe fog and only aggregated data is sent to the cloud.This reduces processing time and allows for knowledgesharing between geographically distributed devices de-ployed in the fog. Machine learning algorithms can beused to update models in the cloud, and then movethem to the fog on a regular basis. This represents amiddle ground between the previously discussed cases.

The usability of the aforementioned deploymentmodels depends, among others, on the type of devicesused in the fog layer, the capability for performinglearning processes, and communication technologies.Thus, flow-based processing for IoT may be analyzed inthe context of the Flow Processing Stack (FPS). FPS isa generic model suited for IoT solutions. It is concep-tually presented in Fig. 3. The stack consists of fourlayers:

• Flows – A data flow can be described as a di-rected graph, where vertices represent compu-tational processes while edges define the flow ofmessages between them. It represents a compu-tation from the business logic perspective withoutdealing with technological aspects related to dataserialization and transmission.

• Ensembles – Ensembles are the executable el-ements that can be deployed and executed inthe execution environments. Depending on their

3

Fog Layer Cloud

Sensor

sensor

ActuatorSensor

max

average

f(x)

Sensor

sensor

ActuatorSensor

max

average

f(x)

Cloud Embedded systems

Sensor

sensor

ActuatorSensor

max

average

f(x)

Sensor

sensor

ActuatorSensor

max

average

f(x)

Embedded systemsFog Layer

Embedded systems

Embedded systems

Sensor

sensor

ActuatorSensor

max

average

f(x)

Sensor

sensor

ActuatorSensor

max

average

f(x)

Embedded systems

Embedded systems

a)

b)

c)

Figure 2: Flow execution examples

4

type, they can assume the form of JSON files forNodeRED software, generated source code thatcan be executed in virtual machines, lightweightcontainers described by dockerfiles or firmwarefor an embedded processor.

• Execution environments – An execution envi-ronment represents a particular execution plat-form, such as an operating system or an appli-cation container that manages the life-cycle ofthe application. Execution environments are typ-ically part of other computing hardware or sys-tems. Execution environment should provide op-erating system-level services, necessary softwarelibraries, required memory and processing poweras needed. In our case, the execution environ-ment is a flow-based processing engine such asNodeRED, uFlow [27] or others.

• Hardware infrastructure – represents the hard-ware infrastructure upon which the execution en-vironments and ensembles are executed. It coversnot only the devices and their capabilities but alsothe network topology and communication tech-nologies.

Hardwareinfrastructure

Flows

Executionenvironments

Ensembles

Figure 3: Flow Processing Stack for IoT

High-level processing flows have to be transformedinto a set of ensembles that can be deployed in theexecution environments. For example, data flow canbe divided into several sub-flows, where some will beexecuted in the NodeRED execution environments in-stalled on various Linux devices and others on uFlowexecuted on embedded devices.

In the paper, we focus on machine learning algo-rithms and how to organize the learning and predictingprocesses. In the next section, the concept of machinelearning architectural patterns will be discussed. Lateron, we analyze latencies introduced by common tech-nologies in the network core and the backhaul that mayhave an impact on latency, thus determining the appli-cability of the system.

4. Machine Learning for IoT

The classic approach to machine learning for IoT as-sumes learning using cloud computing (referred to asBig Data ML). Data from sensors is transmitted to thecloud for analysis over the Internet. Access to histor-ical data stored in a central repository has a numberof advantages as it enables the development of appro-priate machine learning methods to match the needsof specific applications. Unfortunately, centralized dataprocessing also suffers from a number of drawbacks:

• the ever-growing number of devices in the Inter-net of Things generates an increasing amount ofdata that must be transmitted and stored in thecloud;

• lack of autonomy in local subsystems (due to therequirement to transfer data to the cloud for pro-cessing) means that such subsystems cannot per-form their function in the absence of network con-nectivity;

• online systems are usually slow to respond to newevents as they must first upload data to the cloud(which introduces delays);

• there are consumer concerns related to the pro-tection of private data when transferring data tothe cloud or asking for personalized results.

Moving some of the machine learning computationscloser to sensors, i.e. to the network edge (referred asLocal ML) should be transparent to machine learningalgorithms providing results comparable (in terms ofquality) with the Big Data ML approach while retainingthe scalability, personalization, confidentiality, respon-siveness and autonomy characteristic of the Local MLapproach.

In contrast to the stationary problems that can besolved offline using machine learning algorithms, in theIoT domain most of the problems are related to datastreams. These are generated by a number of sen-sors deployed in real-world devices. One of the im-portant problems related to such data is concept drift[28, 29]. It refers to a phenomenon where the tar-get concept changes over time. For example, the be-haviour of customers at a shop may evolve, while ma-chines deployed in factories may change their vibrationcharacteristics due to the wear of their components.These problems can be handled by online learning algo-rithms, where the model is continuously updated withnew data. We distinguish two main groups of such al-gorithms – those that use new data to adjust internalmodels and those that retrain themselves every timenew data is received. The former group does not need

5

to keep all training data in memory, so it is better suitedto streaming data, while the latter stores all trainingdata. Losing et al. analyzed various incremental onlinelearning algorithms [21] and discussed their applicabil-ity to data characterized by concept drift.

Machine Learning

predict

fit

partial_fit

Machine Learning

predict

fit

partial_fit

Figure 4: Machine learning processing block

Fig. 4 depicts the machine learning block that canbe used in the dataflow. The fit input trains the modelbased on the provided data, partial_fit corrects inter-nal models using new data and, finally, predict uses theinternal model to estimate an output value. In the fol-lowing subsection, we discuss the internal structure ofthat block, mindful of the delta architecture concept.

4.1. Delta architecture for machine learning

Delta architecture represents a framework formachine learning algorithms where calculations arespread on the network edge and the cloud. Two dis-tinct architectural layers are defined:

• Collective Intelligence Layer/Central Layer basedin the cloud and used to analyze and process datarepresenting the so-called collective intelligence.This layer enables the discovery of general rela-tionships and trends present in centrally storeddata. It can also use knowledge from servicesavailable on the Internet.

• Edge Layer comprising devices with limited re-sources located on the network edge and usuallyoperating as the Internet gateway for IoT devices.This layer is capable of online sensor data pro-cessing. Due to the limited capabilities of its com-ponent devices, the volume of data that can beprocessed and stored locally is limited.

Fig. 5 depicts the delta patterns for machine learn-ing algorithms. We focus on situations where pre-dictions using estimators should be performed on theedge. In our patterns we distinguish four types of mes-sages:

• S - sensor data;

• S+D - sensor data and the predicted value;

• M - serialized machine learning model;

• D - prediction/decision provided by the model.

The following subsections present three architec-tural patterns of the machine learning processingblock.

4.2. Pattern 0

In pattern P0, data from sensors are aggregatedand transmitted to the cloud where BigData ML mech-anisms generate global models of behaviour based onthe received data. The central layer processes data col-lected from a number of devices and sites. It can adaptto various conditions and creates general models whichare then distributed to the edge layer. Predictions per-formed in the edge layer, based on the sensor data, canbe made locally ensuring short response times. Aggre-gated sensor data can be sent to the cloud at an ap-propriate time optimizing resource consumption [30].In this pattern, learning is only performed in the cloudlayer, while predictions which apply the model occur onthe edge.

4.3. Pattern 1

In the edge learning pattern, sensor data is pro-cessed locally by machine learning modules at theedge. On this basis, a local machine learning modelappropriate for a limited area is trained. Knowledgeis not exchanged across edge locations. In this pattern,machine learning models used on the edge can be set inadvance based on offline learning using historical data.

4.4. Pattern 2

The cloud learning pattern uses the mechanisms ofmachine learning in the cloud layer. Data from sensorsis transmitted to the cloud where BigData ML mech-anisms generate global models of behaviour based onthe received data. Predictions based on the acquiredknowledge are performed in the cloud. The results areaccurate but these patterns introduce the highest pro-cessing latency.

5. Evaluation of delta patterns for machine learn-ing

In this section, we present an experimental eval-uation of the proposed mechanisms. Our goal wastwofold: (i) measure how the delta patterns influencethe accuracy of the common ML models and (ii) de-termine what prediction latency can be achieved usingdifferent back-haul technologies. Comparing both theachieved prediction accuracy and latency using differ-ent patterns allow suggesting pattern suitable for thegiven case as presented in section 5.4.1.

For the purposes of the case study, we prepared syn-thetic data that represent typical situations observed in

6

partial_fit

predict

fit

partial_fit

predict

fit

partial_fit

fit

predict

partial_fit

MLCloud

MLCloud

fit

partial_fit

MLCloud

fit

partial_fit

ML Edge

ML Edge

predict

ML Edge

predict

S

S+D M

predict

fit

partial_fit

predict

fit

partial_fit

Pattern 0 Pattern 1 Pattern 2

D

buffer

partial_fit

predict

fit

partial_fit

predict

fit

partial_fit

cloud

fit

predict

partial_fit

ML Edge

ML Edge

predict

ML Edge

predictS

S+D

D

partial_fit

predict

fit

partial_fit

fit

predict

partial_fit

MLCloud

MLCloud

fit

partial_fit

MLCloud

fit

partial_fit

predict

edge

S

S+D

D

Figure 5: Delta Patterns

the industry. We assumed a single cloud processing in-frastructure and five independent edge environments,e.g. five premises with newly installed product lines orpumps. These devices are equipped with several sen-sors that generate streams of data representing the in-ternal parameters of each device. All devices face thesame problem, i.e. the ageing of their components. Ma-chine learning algorithms are used to predict the stateof the machine – from problem-free operation throughvarious warning states all the way to failures.

The following subsections analyze the hardware in-frastructure of devices that might be deployed on theedge, machine learning algorithms and, finally, back-haul communication technologies.

5.1. Hardware infrastructure analysis

Our case study involves four hardware infrastruc-tures, as presented in Table 2. The table shows howprocessing flows can be mapped to ensembles for agiven hardware infrastructure which is composed ofthe three elements, i.e. sensors, gateways and thecloud (referring to the aforementioned fog computingconcept).

C1 and C2 are the representation of the sample pro-cessing flow from Fig. 2c. In both cases the proposeddelta pattern is P0. In C1, the sensors send data togateways which transmit it to the cloud. The patternthen determines whether to send the updated model tothe edge. The machine learning algorithms on the edgeare deployed on gateways. We have analyzed variousML libraries, such as Weka, Moa and Python Scikit-learn, and finally decided to use the latter in our ex-periments. Our decision was dictated by the fact thatPython is a commonly used programming language forIoT gateways.

In C2, the sensors equipped with GSM connectivityare able to send data directly to the cloud. Since sen-sors have limited resources, they cannot directly run

a Python library used previously to train ML models.In [31] we have proposed the solution that convertsthe ML model to source code and then compiles it intodevice firmware. Table 3 shows how the scikit-learndecision tree model can be transformed to C sourcecode for an embedded microprocessor using the libraryFogML1. It is interesting to note that the Python modelserialized using the Python serialization library calledpickle is 2686 bytes long, while the binary firmware ofthe C code of the predict function compiled using arm-none-eabi-gcc is only 523 bytes long.

C3 refers to the flow depicted in Fig. 2b. In thiscase, the machine learning model is trained only onthe edge, on the gateway device. The recommendedpattern is P1, where learning is performed on thedata available in the particular edge environment. Ofcourse, it is also possible to train ML models using Big-Data algorithms on historical data and then embed thefinal model in the device.

C4 refers to the flow depicted in Fig. 2a. Currently,this is a typical approach used in IoT solutions that relyon machine learning. The model is trained in the cloudon high-performance infrastructure and exposed as aservice for devices located at the network edge. Nev-ertheless, the data from sensors has to be sent to thecloud and predictions have to be sent back to the sen-sors.

Integration of the ensembles deployed in variousexecution environments on different devices might beachieved using one of the IoT communication protocols[32]. Previously we had been relying on the MQTT pro-tocol for that purpose [27].

5.2. Data sets with concept drift

In our experiments, we use stream datasets affectedby concept drift which reflects various physical phe-

1https://github.com/tszydlo/FogML accessed 04.10.2019

7

Table 2: Case study description

C1 C2 C3 C4

cloud scikit-learn software scikit-learn software - scikit-learn software

gateway scikit-learn software - scikit-learn software -

sensors

Firmware for a high-

performance embedded

microcontroller

Sketch generated for

Arduino IDE

Firmware for a high-

performance embedded

microcontroller

Python language script

cloud Python interpreter Python interpreter - Python interpreter

gateway Python interpreter - Python interpreter -

sensors ARM microprocessor Arduino IDE (C based) ARM microprocessorMicroPython embedded

interpreter

sensors-gateway Wi-Fi, Ethernet Wi-Fi, Ethernet

gateway-cloud 2G, 3G, LTE -

cloud Virtual Machine Virtual Machine - Virtual Machine

gatewayEmbedded single-board

computer-

Embedded single-board

computer-

sensors

Embeeded microcontroller-

based device, e.g. ARM

Cortex-M4

ESP8266 board, e.g.

NodeMCU

Embeeded microcontroller-

based device, e.g. ARM

Cortex-M4

An Arduino-compatible

hardware with a mobile

connectivity shield

pattern P0 P0 P1 P2

edge part on gateway on sensors on gateway on sensors

Flow Processing Stack elementsUse Cases – processing approaches

Flows Component-based

Ensembles

Machine Learning

Execution environments

Connectivity2G, 3G, LTE (no

gateway)Wi-Fi

Hardware infrastructure

Table 3: Machine learning model transformation

Python scikit-learn decision tree model

node=0 test node: go to node 1 i f X[ : , 0] <= 3.881 else to node 12.node=1 test node: go to node 2 i f X[ : , 0] <= −0.185 else to node 7.

node=2 test node: go to node 3 i f X[ : , 1] <= −1.662 else to node 4.node=3 leaf node.node=4 test node: go to node 5 i f X[ : , 1] <= 0.975 else to node 6.

node=5 leaf node.node=6 leaf node.

node=7 test node: go to node 8 i f X[ : , 1] <= 0.637 else to node 9.node=8 leaf node.node=9 test node: go to node 10 i f X[ : , 1] <= 4.341 else to node 11.

node=10 leaf node.node=11 leaf node.

node=12 leaf node.

Decision tree estimator for embedded processor

int predict ( f loat * x){i f (x[0]< 3.88166737556) { / / node = 0

i f (x[0] <= −0.185908049345) { / / node = 1i f (x[1] <= −1.66271317005) { / / node = 2

return 3;} else {

i f (x[1] <= 0.975374698639) { / / node = 4return 5;

} else {return 6;

}}

}else {i f (x[1] <= 0.637046217918) { / / node = 7

return 8;} else {

i f (x[1] <= 4.3416762352) { / / node = 9return 10;

} else {return 11;

}}

}} else {

return 12; / / node = 12}

}

nomena. We have two different datasets:

• Circles – dataset consisting of points in two-dimensional space divided into seven categoriesof five hundred points each. The concept drift inthis model is represented by constant-speed rota-tion by π

720×3 per iteration. Each position change,which produces a new point in the stream, causesentire clusters to move.

• RandomTree – dataset generated with the use ofthe MOA data tool2. To emulate concept drift, thedata stream comes from two RandomTree gener-ators with different seeds, used in an alternatingfashion. It is worth noting that this kind of con-cept drift is very slow and weak.

In the experiments, we analyze two scenarios. First,all of the symptoms and failures are observed at eachpremise. In the second scenario, at each premises onefailure category is not observed, but may appear andshould be detected – e.g. a failure that has alreadyappeared at other facilities. The purpose of the sec-ond scenario is to determine how knowledge can beshared between edges via a central processing modulein the cloud. Thus, the stream data expressing physicalphenomena is divided into five streams for each edgepremises. We use two division methods when prepar-ing test data:

• equal – initial data stream is divided in such away that each category is uniformly distributedthrough the edge environments,

2https://moa.cms.waikato.ac.nz/ accessed 04.10.2019

8

• without-one – stream division causes one cate-gory to be missing in each edge environment.

The without-one method was used in order to sim-ulate the situation when none of the edge devices hasfull knowledge of the data set characteristics. In thiscase, only the global model trained on the data comingfrom all of the devices has the ability to recognize allcategories.

5.3. Evaluation of machine learning patterns

We analyzed several machine learning algorithmsand finally decided to use the Decision Tree Classifier,Support Vectors Machine with RBF kernel and Gaus-sian Naive Bayes. One of the deciding factors was theprocessing time of algorithms on embedded devices.

Due to the fact that the data stream is affectedby concept drift, the trained model has to follow thechanges. One of the possible solutions is to train mod-els in a moving frame regime. In this case, the oldestdata is dropped from the frame and new data is insertedas soon as it appears in the data stream. In our experi-ments, we decided to retrain ML models whenever thecontent of the moving frame changes. At each edge fa-cility, the procedure applied when a new point arrivesin the stream is as follows:

1. The point is classified with the currently assem-bled learning model. A score is calculated, whichevaluates to 0 if the point has been misclassifiedand 1 otherwise. The average score for the mostrecent fifty samples is calculated to evaluate theaccuracy of the model.

2. The point is added to the training set (movingframe) located in the cloud (pattern P0 or P2)or locally (pattern P1) only in the equal divisionmethod scenario or when it is not selected to beabandoned in the without-one scenario for theparticular edge.

Table 4 presents results achieved in experimentsconducted for patterns P1 and P2. The size of the win-dow on which the model is trained was 50, 150 and 300respectively. For each case, the average model size andaverage score were calculated for the next 50 consecu-tive samples. Fig. 6 presents charts summarizing gath-ered data.

The following main conclusions can be drawn:

• For pattern P1 and both datasets, the equal di-vision case yields clearly better results than thewithout-one case. This occurs due to the fact thatstoring one common training model in the cloudallows gathering more knowledge of data comingfrom many categories.

• For pattern P2 and both datasets, the results weresimilar regardless of the division method. This isdue to the fact the machine learning is performedcentrally in the cloud.

• For data characterized by stronger concept drift(Circles dataset), moderate frame sizes yieldedthe best results. Lower scores for shorter framesare caused by the training set being too small,while longer frames cause data to retain greaterconcept drift and therefore reduced separability.

Table 5 presents results obtained for pattern P0when the edge model changes every 150 iterations. Themodel score was calculated for the first 50 classifica-tions after each model change, and similarly for furtherclassifications. It can be noted that two datasets be-have very differently depending on the frame offset af-ter the model change. For the dataset burdened withhigher concept drift (Circles dataset), results obtainedduring the first fifty iterations after the change are sig-nificantly better than in the following two frames. Thisis clearly caused by relying on more up-to-date trainingdata, reducing the effects of concept drift. The greaterthe offset from the model change, the less up to datethe model is. For the dataset with a lower concept drift(RandomTree), this phenomenon cannot be observed.It is also worth noting that model size remains constantregardless of the offset chosen.

It is interesting to note that patterns P1 and P2 yieldbetter results than P0 for the equal division method ofthe data streams i.e. when knowledge transfer is notnecessary. In the second case with the without-one di-vision method, P2 gives more accurate results than P0,and P0 gives more accurate results than P1. In P0 andP1 the decision made by the machine learning model isreached directly on the edge; thus response times aresignificantly lower than in P2 where data has to be sentto the cloud.

Our experiments show that it is possible to use ma-chine learning algorithms in edge computing for non-stationary phenomena such as ageing of machines, andachieve short response times with moderate accuracy.The final results depend strongly on data distributionon the network edge. When processes are similar ineach independent edge environment, local edge pro-cessing is sufficient. Nevertheless, when the edge en-vironments need to exchange knowledge, cloud-basedmachine learning becomes mandatory. We have alsoshown that estimators trained using high-level librariescan be transformed into source code for embedded pro-cessors that can be flashed to their internal memory.By applying technologies such as OTA (over-the-air-programming) these processors can be reprogrammedat runtime.

9

Table 4: ML Patterns evaluation - P1 and P2

Model size Score Model size Score Model size Score

tree 2591,259 0,804 2989,894 0,925 3510,553 0,914

svm 3362,118 0,975 4756,824 0,982 6541,059 0,975

bayes 897,000 0,898 897,000 0,967 897,000 0,918

tree 2345,722 0,808 2596,263 0,816 3000,718 0,737

svm 2978,337 0,818 3931,773 0,837 5231,639 0,845

bayes 833,000 0,767 833,000 0,825 833,000 0,782

tree 1656,716 0,725 1973,598 0,749 2100,802 0,802

svm 2423,229 0,545 3756,533 0,692 5170,522 0,745

bayes 904,059 0,714 986,882 0,773 993,000 0,788

tree 1444,506 0,512 1656,359 0,533 1714,273 0,606

svm 1803,471 0,469 2116,976 0,569 2632,359 0,588

bayes 774,176 0,537 851,824 0,573 861,000 0,598

tree 2584,435 0,800 3122,788 0,916 3544,059 0,906

svm 3455,529 0,973 4774,706 0,994 6568,706 0,976

bayes 897,000 0,900 897,000 0,961 897,000 0,914

tree 2607,400 0,841 3046,741 0,910 3605,047 0,910

svm 3391,882 0,975 4644,000 0,994 6435,647 0,980

bayes 897,000 0,880 897,000 0,951 897,000 0,920

tree 2204,765 0,804 2452,608 0,863 3158,490 0,880

svm 4390,000 0,765 8355,667 0,804 13614,000 0,784

bayes 993,000 0,706 993,000 0,843 993,000 0,804

tree 2204,765 0,804 2452,608 0,863 3158,490 0,880

svm 4390,000 0,765 8355,667 0,804 13614,000 0,784

bayes 993,000 0,706 993,000 0,843 993,000 0,804

equal

without-one

RandomTree

equal

without-one

equal

without-one

RandomTree

equal

without-one

Pattern Dataset Division method ML method

Window size

50 150 300

P1

Circles

P2

Circles

Figure 6: Summary of the evaluation of patterns P1 and P2 for RandomTree and Circles datasets.

10

Table 5: ML Patterns evaluation - P0

Model size Score Model size Score Model size Score

tree 3100,200 0,918 3100,200 0,872 3100,200 0,818

svm 4746,440 0,994 4698,320 0,994 4685,720 0,984

bayes 897,000 0,906 897,000 0,914 897,000 0,846

tree 3004,200 0,912 3004,200 0,838 3004,200 0,798

svm 4719,800 0,992 4634,480 0,992 4717,052 0,992

bayes 897,000 0,928 897,000 0,906 897,000 0,822

tree 3353,000 0,860 3353,000 0,890 3353,000 0,940

svm 13270,000 0,800 13270,000 0,840 13270,000 0,840

bayes 993,000 0,800 993,000 0,860 993,000 0,920

tree 3353,000 0,860 3353,000 0,888 3353,000 0,940

svm 13270,000 0,800 13270,000 0,840 13270,000 0,840

bayes 993,000 0,800 993,000 0,860 993,000 0,920

RandomTree

equal

without-one

ML method

Offset after model change

0-49 50-99 100-149

Circles

equal

without-one

Dataset Division method

5.4. Analysis of communication technologies

Data transmission capabilities play an importantrole in the operation of multi-level edge-cloud architec-tures. We have evaluated several transmission mediain terms of applicability to the use case of transferringML models and control-measurement data. We chosemedia commonly utilized and accepted in the indus-try: IEEE 802.3 100BASE-TX Ethernet, 802.11 Wi-Fi,as well as 2G and 3G mobile communication.

The conducted tests evaluate the applicability ofeach medium for the C1-C4 use cases and the cor-responding P0-P2 Delta Patterns described in this pa-per. The data transmitted and received during the testsis synthetic but representative of actual scenarios inwhich sensor data or ML models are sent over a se-lected medium.

As previously mentioned, adequate communicationresponse times depend on the purpose of each system.Overall communication efficiency depends on severalfactors, most importantly network response time andtransfer speed (transmission time of a given amount ofdata). The energy efficiency of the edge-layer nodescan be equally important, because, depending on theuse case, these nodes may be powered with renew-able sources. First, we evaluated and compared the re-sponse times for selected communication standards byanalyzing the first hop response time using the tracer-oute tool. Results are shown in Fig. 7.

A rather obvious factor which significantly affectsthe response time of a connected system is the overalltransmission delay. Further on in this paper, the entiretransfer procedure will be referred to as a transactionwhile the data transfer will be called data transmission.For small chunks of data and responsiveness analysis,the average transmission speed given in e.g. bits/s isless important than the overall transaction time. This iswhy a second evaluation was also performed, focusing

max min

med

ian

aver

age

max min

med

ian

aver

age

max min

med

ian

aver

age

max min

med

ian

aver

age

2G 3G Wi-Fi Ethernet

0.1

1

10

100

1000

10000

Response Times

time

(ms)

Figure 7: First hop response time for selected communication media.

on the transmission times for four different chunks ofdata that correspond to typical use cases in lower lay-ers of the proposed systems. The test data was trans-mitted between the communication device under test(DUT) and a remote server using a TCP-based protocol.Thanks to custom measurement hardware we were ableto separate the connection set-up time (i.e. setting upa connection, disconnecting) and actual data transfer.

2G

Rx

2G

Tx

2G

Rx

2G

Tx

2G

Rx

2G

Tx

2G

Rx

2G

Tx

3G

Rx

3G

Tx

3G

Rx

3G

Tx

3G

Rx

3G

Tx

3G

Rx

3G

Tx

Wi-

Fi R

x

Wi-

Fi T

x

Wi-

Fi R

x

Wi-

Fi T

x

Wi-

Fi R

x

Wi-

Fi T

x

Wi-

Fi R

x

Wi-

Fi T

x

Eth

erne

t R

x

Eth

erne

t Tx

0.0

0.1

1.0

10.0

100.0

0

10

20

30

40

50

60

70

80

90

100

Average transaction time Average transfer time only Data chunk size

Connection type

Tim

e (s

)

Dat

a si

ze (

kB)

Figure 8: Transmission time for selected communication media.

Results of time measurement are presented inFig. 8. The connection set-up time can be very sig-nificant, especially in slow networks, e.g. in 2G mo-bile communications. Ethernet network tests results

11

are shown for the maximum chunk size only due to itsvery low transmission time (resulting in high speed)and insignificant increase in power consumption duringtransmission compared to an idle state. An additionalbut important factor concerns the energy efficiency ofthe communication standard. Fig. 9 presents the com-parison of energy requirements for the same tests as inFig. 8.

2G

Rx

2G

Tx

2G

Rx

2G

Tx

2G

Rx

2G

Tx

2G

Rx

2G

Tx

3G

Rx

3G

Tx

3G

Rx

3G

Tx

3G

Rx

3G

Tx

3G

Rx

3G

Tx

Wi-

Fi R

x

Wi-

Fi T

x

Wi-

Fi R

x

Wi-

Fi T

x

Wi-

Fi R

x

Wi-

Fi T

x

Wi-

Fi R

x

Wi-

Fi T

x

Eth

erne

t R

x

Eth

erne

t Tx

1.00E-03

1.00E-02

1.00E-01

1.00E+00

1.00E+01

1.00E+02

0

10

20

30

40

50

60

70

80

90

100

Transaction energy (J) Data transmission energy only (J) Data chunk size

Connection type

Ene

rgy

[J]

Dat

a si

ze (

kB)

Figure 9: Energy requirements for selected communication media.

Many industrial connected devices such as teleme-try appliances or automated environmental sensors uti-lize the well-established 2G mobile communication net-work. The standard can be used when the local systemis deployed in a place which lacks a direct Internet con-nection or more advanced (3G, LTE) mobile networkcoverage. As can be noted in the presented evalua-tion, the responsiveness of the 2G network is relativelypoor because the transaction time in every 2G trans-fer is longer than one second. This delay would make2G unsuitable for modern industrial requirements (re-fer to Table 1). As in this case computing power ismuch less expensive than data transmission, an obviousneed arises for more advanced computing mechanismswhich do not require any intensive communication withremote cloud servers.

If a relatively slow connection is available, we pro-pose the utilization of Delta Pattern P0 or P1 in whichlocal processing plays a more important role than theavailable connection. Patterns 0 and 1 can also be ap-plied to devices which work in low-rate networks, suchas ZigBee. Additionally, it can be noticed that in P0 atrained model can be transferred to a local device inone of its background tasks, hence transmission timebecomes a less important issue. If a direct Internetconnection is available, the use of Ethernet or Wi-Finetwork is strongly encouraged. This opens new pos-sibilities regarding the availability of critical IoT appli-cations (see Table 1).

Another import aspect related to network commu-nication is access to cloud resources using backbonelinks. We evaluated the latency of several Amazon WebServices (AWS) platform instances, obtaining results

between 45ms for the closest AWS location up to 350msfor farther locations. Based on these results we maychoose the best cloud provider location, offering theshortest latency. The overall latency for sending datafrom the fog to the cloud is the sum of latency intro-duced by the backhaul network technology and the timenecessary to send data over the core Internet links.

5.4.1. Evaluation summary

To conclude, we assigned a recommended connec-tion type for each of the previously mentioned applica-tion areas. A summary is presented in Table 6. As canbe noted, the use of the proposed patterns enables sys-tem designers to consider less demanding connectivityfor applications that would require a faster and moreexpensive connection.

The main differentiating factor is the need forknowledge transfer in the cloud between edge environ-ments. This is expressed in the table as edge/cloudlearning and edge learning only. Another interest-ing observation is that for applications where responsetimes greater than 100ms are acceptable, learning andpredicting in the cloud is sufficient even for the edgelearning case where there is no direct need to transferknowledge. Of course, in the latter case, the edge en-vironment is not autonomous and a persistent Internetconnection is mandatory.

Table 6: Summary of the recommended connection types for eachof the described delta patterns (letters denote connection type: A:Ethernet, B: Wi-Fi, C: 3G or LTE, D: 2G)

PatternRecommended

ConnectivityPattern

Recommended

Connectivity

Factory automation 0.25 - 10 A, B, C, D -

Motion control 1 A, B, C, D -

Tactile Internet 1 A, B, C, D -

Smart Grids 3-20 A, B, C, D -

P0 A, B, C, D -

P2 A -

P0 A, B, C, D -

P2 A, B, C -

A, B, C -

A, B, C -

A, B, C -

A, B, C -

Use caseLatency

(ms)

Edge/Cloud Learning Edge Learning Only

P0

P1

Process automation 50 - 100

Speech recognition 350

P2

Face recognition 500

Intelligent Transport

Systems10 - 100

6. Summary and future work

In this paper, we presented the concept of a flowprocessing stack for IoT that covers the IoT systems onhardware and software layers. We also proposed ar-chitectural patterns which, when applied to machinelearning algorithms, provide the desired quality i.e.short response times and prediction accuracy.

We showed that the decision to apply delta patternswhich involve the distribution of learning process be-tween the edge and the cloud depends on the com-munication technology, proving that data flows can-not be transformed to executable ensembles without

12

prior analysis of the underlying hardware infrastruc-ture and network topology. Additional results includetransfer times and required energy for data sizes of IoTand edge-related tasks, such as sending sensor dataand transferring machine learning models. As futurework, we plan to evaluate the monitoring metrics forhardware infrastructures and sensor data that wouldprompt the use of appropriate delta patterns for ma-chine learning.

Acknowledgements

The research presented in this paper was partiallysupported by the National Centre for Research and De-velopment (NCBiR) under Grant No. LIDER/15/0144/L-7/15/NCBR/2016.

References

[1] V. Rajesh, J. Gnanasekar, R. Ponmagal, P. Anbalagan, Integra-tion of wireless sensor network with cloud, in: Recent Trendsin Information, Telecommunication and Computing (ITC), 2010International Conference on, IEEE, 2010, pp. 321–323.

[2] R. Brzoza-Woch, P. Nawrocki, FPGA-Based Web Services–Infinite Potential or a Road to Nowhere?, IEEE Internet Com-puting 20 (1) (2016) 44–51.

[3] D. Guinard, V. Trifa, Building the Web of Things: With Examplesin Node.Js and Raspberry Pi, 1st Edition, Manning PublicationsCo., Greenwich, CT, USA, 2016.

[4] J. P. Morrison, Flow-Based Programming: A new approach toapplication development, CreateSpace, 2010.

[5] H. Bergius, Noflo–flow-based programming for javascript, URL:http://noflojs. org.

[6] M. Blackstock, R. Lea, Iot mashups with the wotkit, in: Inter-net of Things (IOT), 2012 3rd International Conference on the,IEEE, 2012, pp. 159–166.

[7] M. Hermann, T. Pentek, B. Otto, Design principles for industrie4.0 scenarios, in: System Sciences (HICSS), 2016 49th HawaiiInternational Conference on, IEEE, 2016, pp. 3928–3937.

[8] M. Brettel, N. Friederichsen, M. Keller, M. Rosenberg, How vir-tualization, decentralization and network building change themanufacturing landscape: An industry 4.0 perspective, Interna-tional Journal of Mechanical, Industrial Science and Engineer-ing 8 (1) (2014) 37–44.

[9] E. A. Lee, Cyber physical systems: Design challenges, in: 200811th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC), 2008, pp.363–369.

[10] B. Balis, R. Brzoza-Woch, M. Bubak, M. Kasztelnik, B. Kwolek,P. Nawrocki, P. Nowakowski, T. Szydlo, K. Zielinski, Holistic ap-proach to management of it infrastructure for environmentalmonitoring and decision support systems with urgent comput-ing capabilities, Future Generation Computer Systems 79 (Part1) (2018) 128 – 143.

[11] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean,M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., Tensorflow:A system for large-scale machine learning., in: OSDI, Vol. 16,2016, pp. 265–283.

[12] N. D. Lane, S. Bhattacharya, P. Georgiev, C. Forlivesi, F. Kawsar,Accelerated deep learning inference for embedded and wear-able devices using deepx, in: Proceedings of the 14th AnnualInternational Conference on Mobile Systems, Applications, andServices Companion, ACM, 2016, pp. 109–109.

[13] S. Bang, J. Wang, Z. Li, C. Gao, Y. Kim, Q. Dong, Y.-P. Chen,L. Fick, X. Sun, R. Dreslinski, et al., 14.7 a 288µw pro-grammable deep-learning processor with 270kb on-chip weightstorage using non-uniform memory hierarchy for mobile intelli-gence, in: Solid-State Circuits Conference (ISSCC), 2017 IEEEInternational, IEEE, 2017, pp. 250–251.

[14] H. Sharma, J. Park, N. Suda, L. Lai, B. Chau, V. Chandra, H. Es-maeilzadeh, Bit fusion: Bit-level dynamically composable archi-tecture for accelerating deep neural networks, in: Proceedingsof the 45th Annual International Symposium on Computer Ar-chitecture, ISCA ’18, IEEE Press, Piscataway, NJ, USA, 2018,pp. 764–775. doi:10.1109/ISCA.2018.00069.URL https://doi.org/10.1109/ISCA.2018.00069

[15] A. S. Shamili, C. Bauckhage, T. Alpcan, Malware detection onmobile devices using distributed machine learning, in: Pat-tern Recognition (ICPR), 2010 20th International Conferenceon, IEEE, 2010, pp. 4348–4351.

[16] C. Gupta, A. S. Suggala, A. Goyal, H. V. Simhadri, B. Paran-jape, A. Kumar, S. Goyal, R. Udupa, M. Varma, P. Jain, Protonn:Compressed and accurate knn for resource-scarce devices, in:International Conference on Machine Learning, 2017, pp. 1331–1340.

[17] A. Kumar, S. Goyal, M. Varma, Resource-efficient machine learn-ing in 2 kb ram for the internet of things, in: International Con-ference on Machine Learning, 2017, pp. 1935–1944.

[18] C. Liu, Y. Cao, Y. Luo, G. Chen, V. Vokkarane, Y. Ma, S. Chen,P. Hou, A new deep learning-based food recognition system fordietary assessment on an edge computing service infrastruc-ture, IEEE Trans. Services Computing 11 (2) (2018) 249–261.

[19] T. X. Tran, M.-P. Hosseini, D. Pompili, Mobile edge comput-ing: Recent efforts and five key research directions, MMTCCommunications-Frontiers 12 (4) (2017) 29–34.

[20] G. Peralta, M. Iglesias-Urkia, M. Barcelo, R. Gomez, A. Moran,J. Bilbao, Fog computing based efficient iot scheme for the in-dustry 4.0, in: 2017 IEEE International Workshop of Electron-ics, Control, Measurement, Signals and their Application toMechatronics (ECMSM), 2017, pp. 1–6. doi:10.1109/ECMSM.2017.7945879.

[21] V. Losing, B. Hammer, H. Wersing, Incremental on-line learning:A review and comparison of state of the art algorithms, Neuro-computing 275 (2018) 1261–1274.

[22] S. Yin, X. Xie, J. Lam, K. C. Cheung, H. Gao, An improved incre-mental learning approach for kpi prognosis of dynamic fuel cellsystem, IEEE transactions on cybernetics 46 (12) (2016) 3135–3144.

[23] A. Provodin, L. Torabi, B. Flepp, Y. LeCun, M. Sergio, L. D.Jackel, U. Muller, J. Zbontar, Fast incremental learning for off-road robot navigation, CoRR abs/1606.08057.

[24] F. Bonomi, R. Milito, J. Zhu, S. Addepalli, Fog computing and itsrole in the internet of things, in: Proceedings of the first editionof the MCC workshop on Mobile cloud computing, ACM, 2012,pp. 13–16.

[25] S. Yi, C. Li, Q. Li, A survey of fog computing: concepts, appli-cations and issues, in: Proceedings of the 2015 Workshop onMobile Big Data, ACM, 2015, pp. 37–42.

[26] P. Schulz, M. Matthe, H. Klessig, M. Simsek, G. P. Fet-tweis, J. Ansari, S. A. Ashraf, B. Almeroth, J. Voigt, I. Riedel,A. Puschmann, A. Mitschele-Thiel, M. Muller, T. Elste,M. Windisch, Latency critical iot applications in 5g: Perspec-tive on the design of radio interface and network architecture,IEEE Communications Magazine 55 (2) (2017) 70–78.

[27] T. Szydlo, R. Brzoza-Woch, J. Sendorek, M. Windak, C. Gniady,Flow-based programming for iot leveraging fog computing, in:2017 IEEE 26th International Conference on Enabling Tech-nologies: Infrastructure for Collaborative Enterprises (WET-ICE), 2017, pp. 74–79.

[28] G. Ditzler, M. Roveri, C. Alippi, R. Polikar, Learning in non-stationary environments: A survey, IEEE Computational Intel-

13

ligence Magazine 10 (4) (2015) 12–25.[29] J. a. Gama, I. Žliobaite, A. Bifet, M. Pechenizkiy, A. Bouchachia,

A survey on concept drift adaptation, ACM Comput. Surv. 46 (4)(2014) 44:1–44:37.

[30] T. Szydlo, P. Nawrocki, R. Brzoza-Woch, K. Zielinski, Poweraware mom for telemetry-oriented applications using gprs-enabled embedded devices a levee monitoring use case, in:M. P. M. Ganzha, L. Maciaszek (Ed.), Proceedings of the 2014Federated Conference on Computer Science and InformationSystems, Vol. 2 of Annals of Computer Science and InformationSystems, IEEE, 2014, pp. pages 1059–1064.

[31] T. Szydlo, J. Sendorek, R. Brzoza-Woch, Enabling machine learn-ing on resource constrained devices by source code genera-tion of the learned models, in: Y. Shi, H. Fu, Y. Tian, V. V.Krzhizhanovskaya, M. H. Lees, J. Dongarra, P. M. A. Sloot (Eds.),Computational Science – ICCS 2018, Springer InternationalPublishing, Cham, 2018, pp. 682–694.

[32] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari,M. Ayyash, Internet of things: A survey on enabling technolo-gies, protocols, and applications, IEEE Communications Sur-veys & Tutorials 17 (4) (2015) 2347–2376.

14