A Systematic Mapping Study of Deployment and OrchestrationApproaches for IoT

Phu H. Nguyen1, Nicolas Ferry1, Gencer Erdogan1, Hui Song1, Stephane Lavirotte2, Jean-Yves Tigli2

and Arnor Solberg3

1SINTEF Digital, Oslo, Norway2Universite Nice Sophia Antipolis, Nice, France

3Tellu IoT AS, Asker, Norway{firstname.lastname}@sintef.no {firstname.lastname}@unice.fr {firstname.lastname}@tellu.no

Keywords: IoT, Review, Survey, Deployment, Orchestration, Trustworthiness

Abstract: IoT systems are typically distributed and perform coordinated behavior across IoT, edge and cloud infrastruc-tures. It is very challenging to deploy and orchestrate IoT systems across different layers. We need a clear pic-ture of the research landscape of the existing deployment and orchestration approaches for IoT (DEPO4IOT).Such a picture can show us how advanced the current state of the art is and what are the gaps to address. Weconducted a systematic mapping study (SMS) to find out the research landscape in this area. The results of ourSMS show the overall status of the key artifacts of DEPO4IOT. Among the results, we found a sharp increasein the number of primary DEPO4IOT publications in two recent years. We also found that most approachesdo not really support the deployment or orchestration at low-level IoT devices. There is a lack of addressingthe trustworthy aspects and advanced supports in the existing DEPO4IOT approaches. Finally, we point outthe current open issues in this research area and suggest potential research directions to tackle these issues.

1 INTRODUCTION

The world is now on the verge of “the IoT age”with the Internet of Things (IoT) installed units pre-dicted to grow to 20.4 billion by 20201. IoT systemscan be classified as systems of systems in which phys-ical systems (a.k.a, “things”) and cyber systems arecombined and connected via connection means. De-spite having enormous potential, the heterogeneousnature of IoT brings up great challenges that mustbe addressed to fully realize the potential of the IoT.In particular, because IoT systems are typically dis-tributed and performing coordinated behavior acrossIoT, edge and cloud infrastructures (A. Metzger (Ed.),2015), it is important to facilitate their creation andoperation. In this context, research community andindustry have been proposing different approachesand tools for supporting the deployment and/or or-chestration of IoT systems. However, it is not clearwhat are the primary existing approaches for support-ing the deployment and/or orchestration of IoT sys-tems, and how advanced they are.

To provide a clear picture of the research land-

1Gartner, February 7, 2017

scape of the existing approaches and tools for support-ing the deployment and/or orchestration of IoT sys-tems (DEPO4IOT), we conducted a systematic map-ping study (SMS). Our SMS has three main objec-tives. First, we want to summarize the existing pri-mary DEPO4IOT approaches. Second, by analyzingthe existing approaches, we can identify any gapsin the state-of-the-art. Moreover, we want to ex-amine whether the DEPO4IOT approaches considertrustworthiness aspects such as security, privacy, re-silience, reliability, and safety. Addressing trustwor-thiness aspects is particularly important because IoTsystems can, for instance, have an impact on the phys-ical world through actuators or can deal with sensi-tive data. The ability of a system to evolve, quicklyreact to failures and release patches is decisive to en-sure and increase the trustworthiness of IoT systems.Therefore, we also want to find out if any approachhas provided advanced supports such as monitoringand adaptation. Third, based on the results, we pro-pose new research activities to fill the gaps for sup-porting modern IoT systems. We followed the lat-est guidelines in (Petersen et al., 2015) to conductour SMS. We have systematically filtered thousandsof relevant papers from four main on-line publication

databases, and a manual search process, to finally ob-tain a set of sixty nine (69) primary DEPO4IOT stud-ies. We extracted and synthesized data from the pri-mary studies to answer our research questions (RQs).The main contributions of this work are our answersto the following RQs. RQ1: What are the publi-cation statistics of the primary DEPO4IOT studies?RQ2: What are the primary DEPO4IOT approachesand how advanced are they, especially in addressingtrustworthiness aspects? RQ3: What are the openissues to be further investigated in this field?

In the remainder of this paper: Section 2 givessome background definitions. In Section 3, wepresent our SMS approach. To facilitate the data ex-traction and comparison, Section 4 describes our clas-sification schemes for the primary studies. We presentthe results of our SMS in Section 5. Related work isdiscussed in Section 6. Finally, we conclude the paperwith the major findings and some directions for futurework in Section 7.

2 BACKGROUND

In this section, we give the definitions of IoT sys-tems (subsection 2.1), orchestration and deployment(subsection 2.2), and trustworthiness (subsection 2.3)that were used to define the scope of this study.

2.1 Definition of IoT systems

An IoT system is “a system of entities (includingcyber-physical devices, information resources, andpeople) that exchange information and interact withthe physical world by sensing, processing informa-tion, and actuating” (IEEE Standards Associationet al., 2016). In this work, we consider DEPO4IOTapproaches that are independent or specific to an IoTapplication domain such as smart city or health-care.

2.2 Definition of Deployment andOrchestration

Software deployment typically refers to a post-development activity performed once a piece of soft-ware has been developed to make it available for use(Carzaniga et al., 1998). In this study, approaches areconsidered as supporting deployment when they ex-plicitly offer mechanisms enabling the software de-ployment process, which typically consists of thefollowing stages: (i) release, (ii) installation, (iii)activation, (iv) update, (v) adaptation, and (vi) un-deployment (Dearie, 2007).

Deployment approaches typically rely on the con-cept of software artifacts or components as a mod-ular part of a system that encapsulates its contents(Dearie, 2007). A deployment configuration is thusa connected graph that describes deployment arti-facts along with targets and relationships betweenthem from a structural perspective (Bergmayr et al.,2018). In the cloud context, the term “topology” isoften used as well. A deployment configuration ortopology typically includes the description of how itssoftware components are integrated and communicatewith each other. This typically overlaps with the con-cept of software orchestration (or composition) be-cause an orchestration is also often represented as agraph describing relationships between software ele-ments or processes. However, contrary to deploymentconfigurations, these relationships may represent anorder of operations required to realize a behavior. Us-ing this definition, any service composition approachthat targets IoT domain is considered as an orchestra-tion approach for IoT.

2.3 Definition of Trustworthiness

Trustworthiness refers to the preservation of security,privacy, safety, reliability, and resilience of systems(Griffor et al., 2017). Security refers to the preser-vation of confidentiality, integrity and availability ofinformation (IEC, ISO, 2012). Privacy refers to theprotection of personally identifiable information (PII)(IEC, ISO, 2011). Safety refers to the ability of thesystems to ensure the absence of catastrophic conse-quences on the life, health, property, or data of stake-holders and the physical environment (Griffor et al.,2017). Reliability refers to the ability of the systemsto deliver stable and predictable performance in ex-pected conditions (Griffor et al., 2017). Resiliencerefers to the ability of the systems to withstand in-stability, unexpected conditions, and gracefully returnto predictable, but possibly degraded, performance(Griffor et al., 2017).

3 OUR SYSTEMATIC MAPPINGAPPROACH

We conducted our SMS by following the latestguidelines for conducting SMS in (Petersen et al.,2015) and other relevant guidelines in (Kitchenham,2004). With the context and motivation presented inSection 1, we define our RQs for this work in Section3.1. To explicitly define the scope of our SMS and re-duce possible bias in our selection process, in Section

3.2, we clarify the inclusion criteria for selecting pri-mary studies. Section 3.3 shows our search strategyto find the primary studies for answering the RQs.

3.1 Research questions

We detail the general RQs given Section 1 into thesub-RQs as follows. RQ1 includes three sub-RQs.RQ1.1 - In which years were the primary studies pub-lished and how many publications per year? Answer-ing this question allows us to know for how long thisfield has been started, and how often are there publica-tions. This is an indicator to assess if this field has gotmore attention from the research community. RQ1.2 -In which targeted venues (e.g., Software Engineering-SE, IoT, Cloud/SOA, and Network) and venue types(i.e., conference, journal, workshop) were the pri-mary studies published? Answering RQ1.2 wouldenable us to know which venues have been the tar-gets for publication of primary studies, knowing thatDEPO4IOT approaches could crosscut several relatedresearch areas such as SE, IoT, SOA, Cloud. Thevenue types could also provide some hints on the ma-turity of the primary studies, i.e., papers published atjournals are supposed to report more mature studiesthan papers published at conferences and workshops.RQ1.3 - What is the distribution of publications interms of academic and industrial affiliation? A paperis classified as academia if all authors come from uni-versity or research institute, and industry if all authorscome from a company, and both (academia and indus-try) if there is a mix of authors from academia and in-dustry. Answering RQ1.3 would give us an indicationof how much collaboration between academia and in-dustry on this topic as well as on the interest and needfor DEPO4IOT approaches in the industry.

RQ2 is broken down into the following sub-RQs.RQ2.1: How do the primary DEPO4IOT approachessupport IoT orchestration? Answering RQ2.1 allowsus to assess the characteristics of the primary studiesin IoT orchestration techniques. RQ2.2: How do theprimary DEPO4IOT approaches support IoT deploy-ment? Answering RQ2.2 allows us to assess the char-acteristics of the primary studies in IoT deploymenttechniques. When answering RQ2.1 and RQ2.2, wealso examine how the primary DEPO4IOT approachessupport for deployment, orchestration of IoT systemsacross the vastly heterogeneous IoT, edge and cloudspace. Particularly, we examine how trustworthi-ness aspects are supported by the primary DEPO4IOTstudies. Trustworthiness aspects are important in thedevelopment and operation of IoT systems and arethus often checked in surveys of IoT (e.g., (Tran et al.,2017)). RQ2.3: Are there any primary approaches

explicitly addressing trustworthiness aspects? An-swering RQ2.3 allows us to examine the support ofthe primary DEPO4IOT approaches towards trustwor-thy IoT systems. RQ2.4: Are there (runtime) adap-tation or shared access to resources mechanisms sup-ported in the primary DEPO4IOT approaches? An-swering RQ2.4 allows us to check if any approacheshave provided advanced supports such as adaptationor shared access to resources to increase IoT systemstrustworthiness (see Section 4.3). RQ2.5: How werethe primary DEPO4IOT approaches evaluated? An-swering RQ2.5 allows us to know what evaluationmethods have been used in the primary studies, e.g.,industrial case studies or empirical studies. How a re-search approach has been evaluated is an indicator ofthe maturity of the work, especially if its evaluationused industrial case studies or empirical studies.

Based on the answers to the RQ1 and RQ2, wecan point out the open issues that would deserve moreinvestigation in the future and some potential direc-tions to tackle these issues. RQ3 has two sub-RQs:RQ3.1 - What are the open issues of DEPO4IOT re-search? RQ3.2 - What research directions could berecommended for tackling the open issues?

3.2 Inclusion and exclusion criteria

Based on the RQs and the scope of our study pre-sented in Section 1, we clearly predefined the inclu-sion and exclusion criteria to reduce bias in our pro-cess of search and selection of primary studies. Theprimary studies must meet ALL the following inclu-sion criteria (IC):- (IC1) Must propose a deployment OR orchestrationapproach.- (IC2) Must be explicitly for IoT area, either in gen-eral or in a specific application domain of IoT.- (IC3) Must have software engineering approaches assoftware is the main drive of deployment and orches-tration for IoT.

We excluded non-peer-reviewed or unpublishedpaper, white paper, technical report, thesis, patent,general web page, presentation, book chapter, and pa-per not written in English.

3.3 Search and selection strategy

Fig. 1 shows an overview of the search and selectionprocess with the results for each step, which we de-scribe as follows.

3.3.1 Database search

Using the online search functions of popular publica-tion databases is the most common way to search for

primary studies when conducting secondary studies(Petersen et al., 2015). We used four popular publi-cation databases IEEE Xplore2, ACM DL3, ScienceDirect4, and Scopus5 to search for candidates of pri-mary studies. We did not use Google Scholar andSpringerLink. Scopus and ACM DL already indexSpringerLink6 (Tran et al., 2017). Google Scholar re-turns all kinds of papers, in which peer-reviewed ar-ticles should have been covered by our four chosendatabases. Worse, Google Scholar also returns manynon-peer-reviewed and non-English papers, whichshould have been excluded at the first place. The fourchosen databases contain peer-reviewed articles, andprovide advanced search functions, especially searchin meta-data such as title, abstract, keywords that weused. Based on the RQs, we identified the search key-words. Basically, we used the following search query:(“Internet of Things” OR IoT OR “Web of things” ORWoT) AND (orchestration OR deployment OR chore-ography OR topology OR composition OR dataflow)AND (Tool OR Middleware OR Service OR Frame-work). For each database, the search string needs tobe applied according to the search functions provided.

For each candidate paper, we first read the paper’stitle, keywords and abstract. If a candidate paper ap-pears in more than one database, we only kept theoriginal one where it was published first. If the title,keywords and abstract are insufficient for us to de-cide to exclude a candidate paper, we further skimmedand scanned the paper’s full content. We rather keptany candidate paper in doubt at one point for furtherchecks later. In the end, we hold discussions amongreviewers to crosscheck the candidate papers in doubtand agreed on including or excluding each paper. Af-ter group discussions, we obtained 63 primary studies.

3.3.2 Manual search

It is impractical to make sure the database searchresults can cover all DEPO4IOT publications inour study. Therefore, we tried to complement thedatabase search by a manual search. We started ourmanual search by initiating a set of the DEPO4IOTstudies that we have known of such as the studiesnumbered 22, 24, 26, 30, 31, 35, 42, 49, 50, 53,55 in Table 1. This was also the test set to fine-tune the search query in the database search. More-

2http://ieeexplore.ieee.org3https://dl.acm.org4https://www.sciencedirect.com5https://www.scopus.com6https://www.springer.com/

gp/computer-science/lncs/information-on-abstracting-and-indexing/799288

over, we checked the latest work of the authors ofthese DEPO4IOT studies and their related work tofind more DEPO4IOT studies (using Google scholar).We found six new primary studies that have not beenfound in the database search (because the commonkeywords are not in their titles or abstracts). In total,we obtained the final set of 69 primary studies7 fordata extraction and synthesis to answer the RQs.

Figure 1: Overview of the search and selection steps

4 A TAXONOMY FORCLASSIFICATION

To analyze the primary studies for answering ourRQs, we have defined a taxonomy as a classificationand comparison framework for the primary studies.Our taxonomy consists of the key aspects of deploy-ment and orchestration for IoT (Fig. 2). We groupthem into the following categories: deployment andorchestration support (Section 4.1), design support(Section 4.2), and advanced support (Section 4.3).

4.1 Deployment and orchestrationsupport

4.1.1 Deployment support

Different deployment approaches can be classifiedby deployment kind, target, network specification,whether cloud provisioning is supported, and whatkind of bootstrap is used in deployment process.

Deployment kind: Automatic deployment typi-cally comes in two flavors: imperative and declarative(Binz et al., 2013). The imperative approach requiresa “deployment plan” that details how to reach the de-sired deployment, usually using a workflow language.This deployment plan defines a sequence of atomic

7Our search and selection process for the primary stud-ies ended on 16 March 2018

Figure 2: A Taxonomy of Deployment and Orchestration for IoT

tasks to execute possibly in parallel. This imperativeapproach gives full control over the deployment pro-cess, but it remains difficult to specify and reuse suchplans. In contrast, the declarative approach only re-quires a specification of the desired deployment. Thisstate usually captures the needed application struc-ture and deployment configuration using a domain-specific languages (DSL) and a “deployment engine”then computes how to reach this state. This approachbetter supports evolving and reusing topology mod-els, however the deployment engine may not computeoptimal plans.

Target: IoT systems are typically running overheterogeneous infrastructures ranging from tiny de-vices over gateways to cloud resources. Followingthe layered architecture presented in (Bonomi et al.,2012) we divide this infrastructure into three layers,starting from the most constrained: (i) the devicelayer, which typically consists of embedded devices,sensors and actuators with low power and bandwidth;(ii) the edge/fog layer, which consists of the infras-tructure located at the edge of the network who bridgethe gap between the tiny devices and the cloud; and(iii) the cloud layer, which offers a virtually infiniteset of computing and storage resources. The latter iscommonly categorized based on the layers of virtual-ization (Armbrust et al., 2010): infrastructure (IaaS),platform (PaaS), and software (SaaS).

Network: Orchestration and deployment ap-

proaches must consider networking aspects to ensurethat connected software components, deployed overthe whole IoT, edge and cloud space, can properlyinteract with each other (e.g., custom addressing andsegmentation of launched Virtual Machines - VMs).

Cloud provisioning: One key characteristic of amodern cloud environment is the capability to dynam-ically provision cloud resources (Mell and Grance,2001). Cloud users can provision and release cloudservices on demand and pay only for what they haveactually consumed. This includes compute, storageand network resources. It is not only important todistinguish tools that actually support cloud resourcesprovisioning from others but also to identify whichvirtualization layer (i.e., IaaS, PaaS, SaaS) their pro-visioning engine supports.

Bootstrap: This characterizes the kind of boot-strap the solution relies on. A bootstrap is a basicexecutable program on a device, or a runtime envi-ronment, which the system in charge of the deploy-ment relies on (e.g., Docker). This aspect relates tothe specificity of the solution and its dependency toparticular technologies (Arcangeli et al., 2015).

4.1.2 Orchestration support

Orchestration approaches can be classified by kindsof communication and integration.

Communication support: we classify five dif-ferent kinds of communication support in orches-

tration (Eugster et al., 2003): MessagePassing,SharedSpaces, MessageQueue, Publish-Subscribe,and RemoteInvocations (e.g., Java RMI, CORBA).MessagePassing: the producer sends messagesasynchronously through a communication channel.The consumer receives messages by listening syn-chronously on that channel. Shared space: Produc-ers insert data asynchronously into the shared space,while consumers read data synchronously. Messagequeuing: Messages are stored in a FIFO queue.Producers append messages asynchronously at theend of the queue, while consumers dequeue themsynchronously at the front of the queue. Publish-Subscribe: subscribers have the ability to expresstheir interest in an event, or a pattern of events, andare subsequently notified of any event, generated by apublisher, which matches their registered interest.

Integration support: The interactions betweenthe software components that compose an orchestra-tion or a deployment can be specified and managedat different levels of abstraction. Deployment toolstypically manage logical ports (e.g., port 22 should beopen for SSH) for setting up the communication chan-nel between the software components to be deployed.Orchestration tools, which are more concerned withthe business logic of an application, typically managethese interactions at a lower level of abstractions (e.g.,remote methods invocation).

4.2 Design support

This section presents the design support aspects thatare common for both deployment and orchestration,e.g., specification language, specification capabilities.

4.2.1 Specification language

To specify the orchestration and deployment of an IoTsystem, a language must be used.

Representation: The abstract syntax of a lan-guage defines its concepts and how they relate to eachother. A concrete syntax, is concerned with the form(Moody, 2009) of a language and defines how abstractelements are realized in a concrete representation. Alanguage may have textual and/or graphical syntaxes.

Language type: Languages for the deploymentand orchestration of IoT can be a general purpose lan-guages (GPL) but are typically designed specificallyfor that purpose (i.e., DSL) (Fowler, 2010). A DSL iseither developed on top of an existing GPL (base el-ements of the GPL are extended) or from scratch andhas its own custom concepts without explicit relation-ships to any existing language (Mernik et al., 2005).

Programming model: The selection of a pro-gramming model typically depends on the pragmatic

of a tool. For instance, reactive and event-driven pro-gramming is often used to design IoT systems, flow-based programming is well suited for orchestratingdata flows between IoT devices and services whilstcomponent oriented programming is typically used tospecify deployment models.

4.2.2 Specification capabilities

Here, we want to analyze how the DEPO4IOT ap-proaches specify IoT applications and their corre-sponding deployment structure.

Application structure: To ensure separation ofconcerns, complex distributed systems are typicallydesigned as a set of software components. A com-ponent is basically a unit of computation in an appli-cation, whereas interactions among components arerepresented by connectors (Medvidovic and Taylor,2000). Components and connectors are used to de-scribe the high-level structure of an application interms of a component configuration. Practically, inthis paper we call such components: entities. This isto avoid any confusion with the programming modelused to implement them (e.g., service, component).The application structure may also leverage prede-fined operators like Business Process Model and No-tation (BPMN) operators for orchestration logic.

Deployment structure: Modelling a deploymentstructure typically overlaps with application structurebecause to specify the deployment of a system on a se-lected target infrastructure, its entities need to be allo-cated to IoT, edge or cloud resources. More precisely,what needs to be allocated on these resources are theimplementations of those entities (Bergmayr et al.,2018). The notion of a deployable artifact supportsexactly the reference between logical components andconnectors to their implementations. For instance,a cloud application implemented in Java is possi-bly packaged into several archives, i.e., “JAR files”.Those archives can be represented on the model levelvia artifacts. Allocating them to a cloud service, e.g.,a compute service including a Java platform, shouldhave the effect that the “JAR files” are physically al-located to the provisioned service (Bergmayr et al.,2018). Furthermore, we use the term dependencies todepict the fact that application entities (i) may need tointeract with each other and (ii) may depend on otherentities of specific platforms (Bergmayr et al., 2018).To actually ensure that connected cloud services canin fact interact with each other, properties related tonetworking concerns need often to be explicitly spec-ified (Bergmayr et al., 2018).

4.3 Advanced support

Modern IoT systems must be trustworthy to re-ally realize their values. We check whether theDEPO4IOT approaches address trustworthiness as-pects (presented in Section 2.3), which often requireadvanced supports such as adaptation, monitoring,and shared access to resources.

Adaptation: (McKinley et al., 2004) define twomain types of adaptation: (i) static adaptation and (ii)dynamic adaptation. Static adaptation typically hap-pens at development time whilst dynamic adaptationhappens at runtime. Orchestration and deploymenttools can support these two types of adaptation formodifying: (i) the application itself (e.g., replacingone software component by another) and (ii) its in-frastructure (e.g., bursting from one cloud to another).

Monitoring: Monitoring is a key activity to rea-son on the state of a system and for controlling andmanaging hardware as well as software infrastruc-tures (Aceto et al., 2013). Monitoring probes are usedto deliver information and indicators characterizingthe system and the context in which it is running. Thepurpose of the monitoring can be multiple. In thisstudy we identify the following: (i) measuring QoS ofthe system, (ii) capturing the status and health of a de-ployment, (iii) depicting the state of the environment,and (iv) observing the execution flow of the system,for instance for debugging purpose.

Shared access to resources (direct/indirect):Because IoT systems may involve actuators it is im-portant to control the impact these actuators can haveon the physical world and to manage conflicting ac-tuation requests. More generally, this applies to themanagement of shared accesses to resources, whichcan be of two types: (i) direct concurrent access toresources (e.g., several entities accessing to the sameservice/actuator) or (ii) indirect shared access to a re-source (e.g., actuators from different applications aremodifying the temperature with possibly conflictinggoals) (Ma et al., 2016).

5 RESULTS

Table 1 shows an overview of the primaryDEPO4IOT studies. Based on the taxonomy, we haveextracted and synthesized the data from the primarystudies to answer the RQs of this work.

5.1 Answering RQ1

Answering RQ1.1: as we can see in Fig. 3, the earli-est primary study was published in 2008, when IoT

research was starting to emerge. There is a sharprise in the number of DEPO4IOT publications in thelast two years (2016, 2017), especially regarding thenumbers of journal (J) and conference (C) papers(2016: 3J, 6C and 2017: 7J, 18C). This rise showsthe crucial need of DEPO4IOT research and more at-tention to this research area has gained from the re-search community. We completed our search processin March 2018 and found five primary studies pub-lished in 2018 (3J, 2C).

IoT with its heterogeneous nature spans multiplerelevant research domains among which we identi-fied Software Engineering (SE), Cloud or Service-Oriented Architecture (SOA), Network, and recentlyspecialized IoT research domain (Borgia et al., 2016).Answering RQ1.2: Fig. 4 shows the times that eachresearch domain appears in the calls for papers inthe publication venues where the primary DEPO4IOTstudies published are quite close (SE: 22, IoT: 32,Clould/SOA: 26, Network: 18). Note that the publica-tion venues can have not only one but several researchdomains in their calls for papers. These numbers doreflect the heterogeneous nature of IoT research, withIoT research domain is getting more visible. Dueto the increasing popularity of the IoT research do-main, and because the tools for the deployment andorchestration of application and services in the cloudhave lately reached a high level of maturity, the focusfor the research on deployment and orchestration ap-proaches has moved toward the IoT and Edge spaces.

Figure 3: Publications per year, per venue type

Answering RQ1.3: By checking the affiliationsof the authors, we can see in Fig. 5 that a major-ity of the authors publishing results on DEPO4IOTare academics (86%). The involvement of industryin this research is still very limited, which is under-standable for a relatively new research area like IoT.Answering RQ2.5: Regarding the types of case stud-ies used for evaluating the DEPO4IOT approaches,Fig. 6 also shows the similar dominance of aca-demic approaches. We classify the case studies that

Table 1: Overview of the primary DEPO4IOT studies(sorted by year of publication)

# Title* Year v f1 Challenges and Solutions in Fog Computing Orches-

tration2018 J O

2 Deploying Edge Computing Nodes for Large-scaleIoT: A Diversity Aware Approach

2018 J D

3 Cloud-Fog Interoperability in IoT-enabled HealthcareSolutions

2018 C D

4 A visual programming framework for distributed In-ternet of Things centric complex event processing

2018 J B

5 Enhancing Middleware-based IoT Applicationsthrough Run-Time Pluggable QoS ManagementMechanism

2018 C D

6 A Dynamic Module Deployment Framework forM2M Platforms

2017 C D

7 A Middleware for Mobile Edge Computing 2017 J B8 A service orchestration architecture for Fog-enabled

infrastructures2017 C O

9 Distributed Orchestration in Large-Scale IoT Systems 2017 C O10 Internet of Things: From Small- to Large-Scale Or-

chestration2017 C O

11 QoS-Aware Deployment of IoT ApplicationsThrough the Fog

2017 J D

12 Service Orchestration in Fog Environments 2017 C O13 Towards Container Orchestration in Fog Computing

Infrastructures2017 C O

14 A Framework based on SDN and Containers for Dy-namic Service Chains on IoT Gateways

2017 W D

15 A framework for MDE of IoT-Based ManufacturingCPS

2017 C O

16 A Novel Service-Oriented Platform for the Internet ofThings

2017 C O

17 Design and Implementation of a Message-ServiceOriented Middleware for Fog of Things Platforms

2017 C O

18 Empowering End Users to Customize their Smart En-vironments

2017 J O

19 Feasibility of Fog Computing Deployment based onDocker Containerization over RaspberryPi

2017 C B

20 Semantics Based Service Orchestration in IoT 2017 C O21 An Object-Oriented Model for Object Orchestration 2017 C B22 A TOSCA-based Programming Model for Interacting

Components of Automatically Deployed Cloud andIoT Applications

2017 C B

23 An edge-based platform for dynamic Smart City ap-plications

2017 J D

24 Calvin Constrained: A Framework for IoT Applica-tions in Heterogeneous Environments

2017 C B

25 InterCloud Communication Through Gatekeepers toSupport IoT Service Interaction in the ArrowheadFramework

2017 J O

26 Internet of things out of the box Using TOSCA forautomating the deployment of IoT environments

2017 C B

27 Platform-as-a-service gateway for the Fog of Things 2017 J B28 Runtime deployment and management of CoAP re-

sources for the internet of things2017 J D

29 Composing Continuous Services in a CoAP-basedIoT

2017 C B

30 Niflheim: An end-to-end middleware for applicationson a multi-tier IoT infrastructure

2017 C B

31 Foggy- A Framework for Continuous Automated IoTApplication Deployment in Fog Computing

2017 C D

32 A Web of Things Based Device-Adaptive ServiceComposition Framework

2016 C O

33 Application Orchestration in Mobile Edge Cloud:Placing of IoT Applications to the Edge

2016 W O

34 Optimizing Elastic IoT Application Deployments 2016 J D35 FRED- A Hosted Data Flow Platform for the IoT built

using NodeRED2016 W B

36 Incremental deployment and migration of geo-distributed situation awareness applications in the fog

2016 C D

37 On Building Smart City IoT Applications- aCoordination-based Perspective

2016 W B

38 Orchestrating the Internet of Things Dynamically 2016 W B39 SoIoT: Toward A User-Centric IoT-Based Service

Framework2016 J O

40 A Container-based Edge Cloud PaaS Architecture-based on Raspberry Pi Clusters

2016 C B

41 Automated Deployment of SmartX IoT-Cloud Ser-vices based on Continuous Integration

2016 C D

42 Cloud4IoT: A Heterogeneous, Distributed and Auto-nomic Cloud Platform for the IoT

2016 C B

43 Reliable services composition method for the internetof thing using directed service-object graph deploy-ment scheme

2016 C O

44 Integration of Heterogeneous Services and Thingsinto Choreographies

2016 W O

45 A Scalable Framework for Provisioning Large-ScaleIoT Deployments

2016 J D

46 A Data-Centric Framework for Development andDeployment of Internet of Things Applications InClouds

2015 C D

47 Towards a Semantic Model for Automated Deploy-ment of IoT Services across Platforms

2015 C D

48 A component based approach for the Web of Things 2015 W O49 A Generic Service Oriented Software Platform to De-

sign Ambient Intelligent Systems2015 C O

50 Developing IoT Applications in the Fog: a Dis-tributed Dataflow Approach

2015 C B

51 A Full End-to-End Platform as a Service for SmartCity Applications

2014 W B

52 A Novel Deployment Scheme for Green Internet ofThings

2014 J D

53 glue.things - a Mashup Platform for wiring the Inter-net of Things with the Internet of Services

2014 W B

54 Toward a Distributed Data Flow Platform for the Webof Things

2014 W O

55 BeC3: Behaviour Crowd Centric Composition forIoT applications

2014 J B

56 Dioptase: a distributed data streaming middleware forthe future web of things

2014 J B

57 Application deployment for IoT: An infrastructure ap-proach

2013 C B

58 Orchestration in distributed web-of-objects for cre-ation of user-centered iot service capability

2013 C O

59 Towards Automated IoT Application Deployment bya Cloud-Based Approach

2013 C D

60 Mobile Fog: A Programming Model for Large-ScaleApplications on the Internet of Things

2013 C D

61 Gateway as a service- A cloud computing frameworkfor web of things

2012 C O

62 Behaviour-Aware Compositions of Things 2012 C O63 Knowledge-Aware and Service-Oriented Middleware

for deploying2012 J B

64 Mashing up the Internet of Things: A framework forsmart environments

2012 J O

65 D-LITE: Distributed logic for internet of things ser-vices

2011 C B

66 Adaptable Service Composition for Very-Large-ScaleIoT Systems

2011 W O

67 INOX: A managed service platform for inter-connected smart objects

2011 W B

68 Connecting Smart Things through Web Services Or-chestrations

2010 W O

69 COSMOS: a middleware for integrated data process-ing over heterogeneous sensor networks

2008 J O

PV = Short name of publication venue.vVenue type: J = Journal (19), C = Conference (37), W = Workshop (13).fFocus: D = Deployment, O = Orchestration, B = Both Deployment and Orchestration.* The titles are clickable to link to the corresponding publications

are not from industry as academic ones, e.g., moti-vational examples, prototypes or simulations devel-oped by researchers for discussing or evaluating theirDEPO4IOT approaches. Nearly one tenth of the stud-ies do not really provide evaluation details (not avail-able, N/A) because of the early stage of their work,e.g., reported in short papers or new ideas papers.But, the number of industrial case studies or empir-ical studies (with industry) account for 13% in to-tal, which is still encouraging. We would call formore collaboration between academia and industryfor more practical DEPO4IOT research in particular,but also IoT research in general.

5.2 Answering RQ2: Deployment &Orchestration support

Answering RQ2.1 and RQ2.2: Fig. 7 shows thedistribution of primary studies based on the focus:

Figure 4: Research topics per publication venue

Figure 5: The affiliationsof authors Figure 6: Evaluation case

studies

orchestration, or deployment, or both orchestrationand deployment. We can see that the orchestration-focused studies are nearly double the deployment-focused studies. The main reason could be thatthe orchestration-focused studies are around IoT datamash-up at cloud or edge level, which we show laterin Fig. 8. It is understandable that the first IoT re-search focus is more on building IoT applications (or-chestration) before deploying them. These studies donot technically contribute to low-level orchestrationinvolving IoT devices or gateways but rather makeassumptions on low-level IoT infrastructure. Aboutone-third (37%) of the approaches support both de-ployment and orchestration. Note that the degrees ofsupport for deployment and orchestration are not atthe same level. In other words, approaches that sup-port deployment somehow also support orchestration(as part of the deployment specifications overlap withorchestration specifications). However, the orchestra-tion support in these cases is often at a higher level ofabstraction than the orchestration-focused approachesthat offer specific support for orchestration (i.e., logi-

cal port vs. method).

Figure 7: Main focus of the primary DEPO4IOT studies

Figure 8: Target infrastructure

Fig. 8 shows how the primary DEPO4IOT stud-ies support for different layers of IoT infrastructure:cloud, edge, or IoT devices. Most studies (71%) dis-cuss about mashing up data streaming from IoT de-vices at cloud (7%) or edge/fog (32%) or both (32%).But, few studies (29%) really support orchestratingand/or deploying software on IoT devices. We wouldargue that deployment and orchestration at IoT de-vices are the most challenging research problems inDEPO4IOT because of the diversity of IoT devices,their networking protocols and connectivity issues,and their different computing resource constraints. Toreally support for modern real IoT systems in whichtrustworthy aspects are crucial, DEPO4IOT studiesmust advance to the technical details of edges and IoTdevices. Even among the 29% mentioned above, wefind very limited support for modern IoT systems aswe discuss in the following paragraphs.

Looking closer into the studies that support de-ployment (59%, Fig. 7), we find that nearly two-third(64%, Fig. 9) of them have declarative deploymenttype vs. one-third have imperative type. We would

Figure 9: Deployment en-gine

Figure 10: Cloud provi-sioning

argue this is because imperative approaches are typi-cally more complicated to specify and reuse (as theyrequire the specification of a deployment plan). Onthe other hand, they give full control over the deploy-ment process, thus allowing its optimization. Veryfew approaches (6%) provide users with the abilityto select between both approaches. We would also ar-gue that hybrid approaches could be beneficial. Forinstance, a deployment engine could first derives atentative deployment plan from a deployment config-uration describing the desired system state. Then, ifnecessary, the user or a reasoning engine could use animperative approach to adjust this plan before the ac-tual deployment. Also among 59% of the studies thatsupport deployment, less than half (46%) support theprovisioning of cloud resources before deployment(Fig. 10). This means only one-third (59% times46%) of the total primary studies support cloud re-sources provisioning, which is an important feature inany advanced DEPO4IOT approach. This illustratesthe need for further integration of the works from thecloud and IoT domains.

Among the studies that support orchestration(78%, Fig. 7), we find that the communication typesin orchestrating IoT components are diverse (Fig. 11).However, we observe that approaches offering thebest decoupling in time and space (Eugster et al.,2003) (between subscribers and publisher), i.e., mes-sage passing, message queues and publish/subscribe,are largely adopted (73% in total). The integrationlevel at methods (66%, Fig. 12) for orchestration ismore common than at logical port (34%). Only 11%support both levels, which is often needed for orches-trating more complex IoT systems.

5.3 Answering RQ2: Design support

This section gives more answers to the RQ2.1 andRQ2.2, regarding the design support aspects of or-chestration and deployment. Fig. 13 shows that themost common term to depict deployment and orches-

Figure 11: Orchestration com-munication types

Figure 12: Integra-tion level

tration entities is service, which is significantly moreoften than component and node. We find that mostorchestration approaches have used services for datamash-up. Node term seems getting more common insupporting both deployment and orchestration (e.g.,in NodeRED-based approaches). The total numberof primary studies that provide technical informa-tion about connectors is 43, which is two-third of theDEPO4IOT approaches. High-level operators (likeBPMN operators) for supporting deployment and or-chestration are not common because very few havebeen mentioned in the approaches.

Figure 13: Application structure

Any thorough DEPO4IOT approaches, especiallyfor deployment and/or orchestration at IoT deviceslevel, should have provided technical informationabout bootstrap and network specification. But, Fig.14 shows that only less than one-third (32%) of thedeployment approaches have provided any such in-formation. This could be interpreted that most currentapproaches do not address the technical details of de-ployment and make assumptions on the support of un-derlying operating systems or execution engines. Wealso find in Fig. 14 only about half having either boot-strap (52%) or network specification (55%) with de-ployment. A thorough deployment approach shouldhave included network specification(s) to support forlow-level deployment on IoT devices, where diversenetwork topologies and protocols must be considered.

Figure 14: Bootstrap andnetwork specification pro-vided with deployment?

Figure 15: Programmingmodel

Figure 16: Languagetypes

Figure 17: Representationkinds

Aligned with the popular of “service” discussed inthe previous paragraphs, service-oriented model is themost common (34%, Fig. 15) programming model inthe DEPO4IOT approaches, followed by component-based (19%), and flow-based (13%). These modelscan provide a modular, loosely-coupled specificationof the orchestration and its deployment so that themodules can be seamlessly substituted and reused.

The language support is important to makeDEPO4IOT approaches more practical, but more thanone-third of the approaches do not provide languagesupport for deployment and orchestration (N/A in Fig.16). Using DSL seems to be more popular than GPLbecause DSL could be more expressive in specifyingDEPO4IOT aspects. Fig. 17 shows that textual is themost popular form of language support (28%), com-pared to graphical (17%). Some approaches (12%) dohave both graphical and textual formats.

5.4 Answering RQ2: Trustworthiness &advanced supports

Answering RQ2.3 and RQ2.4: Trustworthiness as-pects must be supported in the orchestration and de-ployment of modern IoT systems. However, Fig. 18shows that very few primary DEPO4IOT studies ad-dress trustworthy aspects: security (18 out of 69 stud-ies), privacy (8), resilience (6), reliability (8). Among

Figure 18: Trustworthiness aspects

Figure 19: Advanced supports

the trustworthiness aspects, security is the most ad-dressed one in the primary DEPO4IOT studies. Evenso, most of 18 DEPO4IOT studies addressing se-curity only briefly mention security aspects in theirDEPO4IOT approaches. Studies that address securityin more details are very rare (e.g., the study #24 in Ta-ble 1). None addresses safety, which should be alsoa crucial property of critical IoT systems. This re-lates to the Shared access to resources (including ac-cess to actuators), which is a rare feature in the ex-isting DEPO4IOT approaches (e.g., in the study #49).We believe a reason for that is that IoT system inno-vations have until now mainly been concerned withsensors, device management and connectivity, withthe mission to gather data for processing and analy-sis in the cloud in order to aggregate information andknowledge8. Management of actuators has thus onlyrecently appeared as a research priority in the com-munity. In general, Fig. 19 shows that few primaryDEPO4IOT studies provide advanced supports suchas monitoring, runtime adaptation, shared access. Itis worth noting that it is often the same approachesthat support adaptation and monitoring. We believethis is due to the fact that both are complementary.

8IEC white paper entitled “IoT 2020: Smart and secureIoT platform”. http://www.iec.ch/whitepaper/pdf/iecWP-loT2020-LR.pdf

Indeed, before triggering an adaptation it is importantto observe and understand the status of the system.

5.5 Answering RQ3

This section gives our answers to the RQ3.1 andRQ3.2 that are supported by the findings presentedabove. Even though there is a big jump in the numberof primary DEPO4IOT studies recently, our analysesshow that DEPO4IOT research is still in its infancy.Thus, there are fundamental technical gaps and openissues to be tackled.

We found that the number of orchestration ap-proaches, often for IoT data mash-up, is nearly dou-ble the number of deployment approaches. The or-chestration approaches for IoT data mash-up such asthe studies numbered 10, 20, 33, 62, 64 in Table 1often make assumptions of readily deployed IoT sys-tems in operation. IoT deployment approaches shouldreceive more attention. Without proper deploymentapproaches to deploy IoT systems, orchestration ap-proaches cannot thrive. Besides, it is worth to notethat most of the primary DEPO4IOT approaches donot really support deployment and/or orchestration atIoT devices, e.g., without bootstrap or network spec-ification. Only by going into low-level details, IoTengineering can really control the whole chain of IoTsoftware deployed from cloud until IoT devices. Inthis way, it would be more likely that the trustworthyaspects and advanced supports can be addressed moresystematically and efficiently.

They are also immature in addressing trustwor-thy aspects and advanced supports. The numberof existing DEPO4IOT studies addressing trustwor-thy aspects is very small. Even among those stud-ies, we have not really found any that explicitly andsystematically considered trustworthy aspects. Thesame observation applies to the advanced support ofDEPO4IOT studies, i.e., regarding monitoring, adap-tation, and shared access to resources. We wouldpropose that new DEPO4IOT research should addressmore thoroughly and systematically the trustworthyaspects and advanced supports, which are crucial formodern IoT systems.

Finally, the dominance of academia-only inDEPO4IOT research suggests that there should bemore collaboration between academia and industryto make DEPO4IOT approaches more practical andcloser to the needs in industry.

6 Related work

IoT has emerged as an important area of researchand development in the recent years. There have beenefforts to review the state of the art in different as-pects of IoT. Some surveys have addressed IoT mid-dleware (Razzaque et al., 2016; Ngu et al., 2017), IoTplatforms (Mineraud et al., 2016), and IoT architec-tural concerns (Gill et al., 2017) but none has sys-tematically, specifically investigated deployment andorchestration approaches for IoT. Note that we haveclearly defined the scope of our SMS, which only con-sidered peer-reviewed publications, not white papersfrom industry. Thus, our SMS reports the state of theart in DEPO4IOT research, not including the state ofpractice in industry.

From the software architecture view, an IoT mid-dleware provides a layer between application soft-ware and system software. DEPO4IOT approachesare not necessary about middleware because theyare more vertical along the IoT engineering life-cycle from development to operation of IoT sys-tems. Research on IoT middleware highly overlapswith DEPO4IOT because DEPO4IOT approaches of-ten leverage middlewares for integrating heteroge-neous computing and communications devices, andsupporting interoperability within the diverse appli-cations and services running on these devices. In(Razzaque et al., 2016) and (Ngu et al., 2017), theauthors conducted two different surveys of the ex-isting middlewares to classify them and find out themain challenges. The results of these studies not onlyaddress the functional aspects of IoT middleware butalso some quality aspects such as security, adaptationthat we also considered in our work. But, these twostudies are not systematic studies.

(Mineraud et al., 2016) provide a gap analysis onthe well-known IoT platforms. In particular, the au-thors identify gaps related security (fine grained ac-cess control), cross-platform and cross-layer DSL toreduce threats on privacy and security. Their workis not a systematic study and mainly focuses on plat-forms maturity and usability. (Gill et al., 2017) con-ducted a systematic study on the key IoT architec-tural concerns. It highlights a set of challenges thatis a combination of technical, human, financial andethical aspects. Their work is complementary to ourwork reported in this paper as it does not focus ondeployment, orchestration and trustworthiness. How-ever, our work and (Gill et al., 2017) share some com-mon findings, e.g., the lack of support for IoT security,and the need for runtime adaptation of IoT systems.

7 CONCLUSIONS

Deployment and orchestration approaches for IoTshould be advanced enough to support distributed pro-cessing and coordinated behavior across IoT, edgeand cloud infrastructures. In this paper, we have ex-amined the research landscape of deployment and or-chestration approaches for IoT, by conducting a sys-tematic mapping study. After systematically identi-fying and reviewing 69 primary studies out of thou-sands relevant papers in this field, we have found outthat 1) there is a sharp rise in the number of publi-cations addressing this field in the two recent years;2) however, there are still different gaps that the cur-rent approaches seem to be immature to address suchas the real, low-level technical details of deploymentand/or orchestration at IoT devices level; 3) new de-ployment and/or orchestration approaches should alsofocus on addressing the trustworthy aspects and ad-vanced supports for modern IoT systems. To make theIoT deployment and orchestration approaches morepractical, there should be more and more researchcollaborations between academia and industry. Wewill address these open issues in our current researchprojects, especially for the trustworthiness of IoT.

REFERENCES

A. Metzger (Ed.) (2015). Cyber physical systems: Opportu-nities and challenges for software, services, cloud anddata.

Aceto, G., Botta, A., De Donato, W., and Pescape, A.(2013). Cloud monitoring: A survey. Computer Net-works, 57(9):2093–2115.

Arcangeli, J.-P., Boujbel, R., and Leriche, S. (2015). Au-tomatic deployment of distributed software systems:Definitions and state of the art. Journal of Systemsand Software, 103:198–218.

Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz,R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A.,Stoica, I., et al. (2010). A view of cloud computing.Communications of the ACM, 53(4):50–58.

Bergmayr, A., Breitenbucher, U., Ferry, N., Rossini, A.,Solberg, A., Wimmer, M., Kappel, G., and Leymann,F. (2018). A systematic review of cloud modeling lan-guages. ACM Comput. Surv., 51(1):22:1–22:38.

Binz, T., Breitenbucher, U., Haupt, F., Kopp, O., Leymann,F., Nowak, A., and Wagner, S. (2013). Opentosca–aruntime for tosca-based cloud applications. In Inter-national Conference on Service-Oriented Computing,pages 692–695. Springer.

Bonomi, F., Milito, R., Zhu, J., and Addepalli, S. (2012).Fog computing and its role in the internet of things. InProceedings of the first edition of the MCC workshopon Mobile cloud computing, pages 13–16. ACM.

Borgia, E., Gomes, D. G., Lagesse, B., Lea, R. J., and Puc-cinelli, D. (2016). Special issue on “internet of things:Research challenges and solutions”. Computer Com-munications, 89:1–4.

Carzaniga, A., Fuggetta, A., Hall, R. S., Heimbigner, D.,Van Der Hoek, A., and Wolf, A. L. (1998). A char-acterization framework for software deployment tech-nologies. Technical report, Colorado State Univ FortCollins Dept Of Computer Science.

Dearie, A. (2007). Software deployment, past, presentand future. In Future of Software Engineering, 2007.FOSE’07, pages 269–284. IEEE.

Eugster, P. T., Felber, P. A., Guerraoui, R., and Kermarrec,A.-M. (2003). The many faces of publish/subscribe.ACM computing surveys (CSUR), 35(2):114–131.

Fowler, M. (2010). Domain-specific languages. PearsonEducation.

Gill, A. Q., Behbood, V., Ramadan-Jradi, R., and Beydoun,G. (2017). Iot architectural concerns: a systematicreview. In Proceedings of the Second InternationalConference on Internet of things and Cloud Comput-ing, page 117. ACM.

Griffor, E. R., Greer, C., Wollman, D. A., and Burns, M. J.(2017). Framework for cyber-physical systems: Vol-ume 1, overview. Technical report.

IEC, ISO (2011). Information technology-securitytechniques-privacy framework. International Stan-dard No. ISO/IEC.

IEC, ISO (2012). Information technology-securitytechniques-information security managementsystems-overview and vocabulary. InternationalStandard No. ISO/IEC, 27000:32.

IEEE Standards Association et al. (2016). P2413-standardfor an architectural framework for the internet ofthings (iot). Institute of Electrical and Electronics En-gineers, New York.

Kitchenham, B. (2004). Procedures for performing sys-tematic reviews. Keele, UK, Keele University,33(2004):1–26.

Ma, M., Preum, S. M., Tarneberg, W., Ahmed, M., Ruiters,M., and Stankovic, J. (2016). Detection of run-time conflicts among services in smart cities. InSmart Computing (SMARTCOMP), 2016 IEEE Inter-national Conference on, pages 1–10. IEEE.

McKinley, P. K., Sadjadi, S. M., Kasten, E. P., and Cheng,B. H. (2004). A taxonomy of compositional adapta-tion. Rapport Technique.

Medvidovic, N. and Taylor, R. N. (2000). A classificationand comparison framework for software architecturedescription languages. IEEE Transactions on softwareengineering, 26(1):70–93.

Mell, P. and Grance, T. (2001). The NIST Definition ofCloud Computing. Special Publication 800-145, Na-tional Institute of Standards and Technology.

Mernik, M., Heering, J., and Sloane, A. M. (2005). Whenand how to develop domain-specific languages. ACMcomputing surveys (CSUR).

Mineraud, J., Mazhelis, O., Su, X., and Tarkoma, S. (2016).A gap analysis of internet-of-things platforms. Com-puter Communications, 89.

Moody, D. (2009). The “physics” of notations: toward a sci-entific basis for constructing visual notations in soft-ware engineering. IEEE Transactions on Software En-gineering, 35(6):756–779.

Ngu, A. H., Gutierrez, M., Metsis, V., Nepal, S., and Sheng,Q. Z. (2017). Iot middleware: A survey on issues andenabling technologies. IEEE Internet of Things Jour-nal, 4(1):1–20.

Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015).Guidelines for conducting systematic mapping stud-ies in software engineering: An update. Informationand Software Technology, 64:1–18.

Razzaque, M. A., Milojevic-Jevric, M., Palade, A., andClarke, S. (2016). Middleware for internet of things: asurvey. IEEE Internet of Things Journal, 3(1):70–95.

Tran, N. K., Sheng, Q. Z., Babar, M. A., and Yao, L.(2017). Searching the web of things: state of the art,challenges, and solutions. ACM Computing Surveys(CSUR), 50(4):55.