Systematic Evaluation of Sandboxed Software Deployment forReal-time Software on the Example of a Self-Driving Heavy Vehicle

Philip Masek1, Magnus Thulin1, Hugo Andrade1, Christian Berger2, and Ola Benderius3

Abstract— Companies developing and maintaining software-onlyproducts like web shops aim for establishing persistent links totheir software running in the field. Monitoring data from realusage scenarios allows for a number of improvements in thesoftware life-cycle, such as quick identification and solutionof issues, and elicitation of requirements from previouslyunexpected usage. While the processes of continuous integra-tion, continuous deployment, and continuous experimentationusing sandboxing technologies are becoming well established insaid software-only products, adopting similar practices for theautomotive domain is more complex mainly due to real-timeand safety constraints. In this paper, we systematically evaluatesandboxed software deployment in the context of a self-drivingheavy vehicle that participated in the 2016 Grand CooperativeDriving Challenge (GCDC) in The Netherlands. We measuredthe system’s scheduling precision after deploying applications infour different execution environments. Our results indicate thatthere is no significant difference in performance and overheadwhen sandboxed environments are used compared to nativelydeployed software. Thus, recent trends in software architecting,packaging, and maintenance using microservices encapsulatedin sandboxes will help to realize similar software and systemengineering for cyber-physical systems.

I. INTRODUCTION

Companies that produce and maintain software-only prod-ucts, i.e. products that are only constituted with softwareand are pre-dominantly executed in cloud-environments likeweb shops or cloud-based applications, have started to strivefor achieving shorter product deployment cycles. Their goalis to tighten software integration and software deploymentup to their customers. When these processes are reliablyestablished, companies are able to closely monitor theirproducts in real usage scenarios from their customers andcan collect highly relevant data thereabouts. This data is ofessential business interest to better understand the productsin the field, maintain them in case of issues, and to graduallyintroduce new features and observe their adoption.For example, companies like Facebook and Netflix haveestablished mechanisms to enable continuous deploymentthat would allow software updates as often as hundredsof times a day [1]. These companies ultimately aim for a

1Department of Computer Science and Engineering, Chalmers Universityof Technology, Goteborg sica@chalmers.se

2Department of Computer Science and Engineering, University ofGothenburg, Goteborg christian.berger@gu.se

1Department of Applied Mechanics, Chalmers University of Technology,Goteborg ola.benderius@chalmers.se

We are grateful to Chalmers University of Technology to support theresearch in preparation for the 2016 GCDC competition. The ChalmersRevere laboratory is supported by Chalmers University of Technology, ABVolvo, Volvo Cars, and Region Vastra Gotaland.

Fig. 1. Volvo FH16 truck from Chalmers Resource for Vehicle Research(Revere) laboratory that participated in the 2016 Grand Cooperative DrivingChallenge in Helmond, NL.

software engineering process that integrates near real-timeproduct feedback data as crucial method for continuousbusiness and product innovation.A key-enabling technology for such software-only productsis resource isolation that is strictly separating system re-sources like CPU time, network devices, or inter-processcommunication resources; today’s leading environments arethe tool suite Docker [2], [3] or Google’s lmctfy (from letme contain that for you) [4] that encapsulates cloud-basedapplications. The key concept is to package all relevantsoftware product dependencies in self-contained bundles thatcan be easily deployed, traced, archived, and even safelyrolled back in case of issues.This study is an extension of a master thesis presented in[5], where further details are presented and available.

A. Problem Domain & Motivation

Continuously developing, integrating, and deploying is chal-lenging with software-only systems, i.e., systems that arenot dependent on their surrounding physical environments.However, these tasks become even more challenging inthe domain of self-driving vehicles, where applications aresafety-critical and most of the times, even the underlyinghardware is limited in capacity. In addition, this type ofcyber-physical system (CPS) heavily relies on real-time

arX

iv:1

608.

0675

9v1

[cs

.SE

] 2

4 A

ug 2

016

capabilities, making the scheduling precision a fundamentalaspect to consider throughout the entire lifecycle.This study is mainly motivated by our experience with a self-driving truck depicted in Fig. 1 that participated in the 2016Grand Cooperative Driving Challenge (GCDC)1. The com-petition includes a number of collaborative maneuvers thatmust be safely performed by participating vehicles. As thereare several team members constantly working on the truck’ssoftware, it has become evident that the project would greatlybenefit from a reliable, traceable, and structured deploymentprocess. On the other hand, significant performance overheadin the software execution must be avoided apparently.

B. Research Goal & Research Questions

The goal of this work is to systematically evaluate the influ-ence of sandboxed execution environments for applicationsfrom the automotive domain. We are particularly interested instudying the impact on two quality attributes of the system:Scheduling precision and input/output performance. Hence,in this article the following research questions are addressed:

RQ-1: How does the execution environment influence thescheduling precision of a given application?

RQ-2: How does the execution environment influence theinput/output performance of a given application?

C. Contributions of the Article

We demonstrate the impact of using container-based technol-ogy in relation to the underlying operating system (i.e., real-time kernel) on the performance of a given automotive appli-cation. To achieve this, we conduct a multi-method researchapproach that includes (i) an investigation of the currentstate-of-the-art in the area; (ii) a controlled experiment ondesk using example applications; and (iii) a validation ofthe results in a real-world scenario using the aforementionedVolvo FH truck. To the best of our knowledge, this workis the first of its kind to evaluate the use of Docker in theautonomous vehicles business.

D. Structure of the Article

The remainder of this article is structured as follows. Sec. II,contains the methodology used in this work. In Sec. III,the related work is presented in the form of a literaturereview. Sec. IV contains a description of the experimentsin a controlled environment and in the truck. In Sec. V wepresent the results, followed by an analysis and discussionin Sec. VI. Finally, conclusion and future work are describedin Sec. VII.

II. METHODOLOGY

In order to achieve an understanding of how the execu-tion environment influences the performance of applications,three studies are designed in a way that they complementedeach other. First, we perform a literature review using keyterms in the search, followed by the inclusion of additionalpapers using the snowballing approach [6]. Then, we conducta controlled experiment on desk to measure the scheduling

1http://www.gcdc.net

precision and I/O performance of sample applications whendeployed on four different environments. Finally, we conductan experiment using a real-world application that is currentlydeployed on the self-driving truck. The latter experiment isdesigned in a way that the findings from the first study canbe validated.The goal with such design is to obtain meaningful resultsfrom different sources, combine them, and contribute towardsa safe, robust, and reliable software deployment process inthe domain of self-driving vehicles. Performing such multi-method research allows collection of data of different types,resulting in a wider coverage of the problem space [7].

III. LITERARATURE REVIEW

Performing a literature review in a way that it is systematicbrings a number of benefits. The steps are replicable, jus-tified, and the results of the study provide basis for futureresearch in a given area. In the present work, we performa light-weight literature review based on the snowballingapproach [6]. The technique consists of searching for relatedwork by investigating citations from a known set of relevantpapers. In summary, the steps to conduct a snowballing pro-cedure include (i) selecting a set of papers referred to as thestart set; (ii) apply forward snowballing (citation analysis) inthe start set; and (iii) apply backward snowballing on eachpaper identified. This process iterates until no new relevantpapers are found.Our start set consists of papers found through a search in theScopus digital library [8]. The search string is presented inTable I. It contains key terms concerning performance evalu-ation of virtualization/container approaches. We also includethe term Docker as authors may have different definitionsfor the framework, yet we are interested in capturing papersdiscussing it. The search resulted in 215 papers that hadtheir titles and abstracts critically evaluated according to ourinclusion and exclusion criteria. Inclusion criteria includepapers presenting benchmarking activities and experienceswith Docker. Exclusion criteria exclude, for example, papersin languages other than English and short papers pointing toonline material.The forward and backward snowballing procedures resultedin 10 highly relevant papers which were selected as primarystudies. In addition to this set, we included 2 external papersthat were not found during the search, even though theyare relevant to the context of this study. The final set wascritically appraised in the search for insights that would aidin the understanding of the state-of-the-art of software vir-tualization/containers. The next subsection contains selectedoutcomes from the search.

TABLE ISEARCH STRING IN SCOPUS

TITLE-ABS-KEY(Performance OR Comparison OR LatencyOR Evaluation OR Container-Based OR Linux Containers

OR Lightweight Virtualization OR Container Cloud OR Docker)AND

(LIMIT-TO(SUBJAREA,“COMP”)

A. Outcomes of the Review

The challenge and need for maintaining, deploying, andconfiguring software for embedded, cyber-physical systemsis identified in current literature [9], [10]. Berger [9] investi-gates a continuous deployment (CD) process for self-drivingminiature vehicles using Docker containers. Multiple con-tainers are used to create binary packages from source thatare signed and tested for different hardware architectures,exemplifying a possible CD process for CPSs.The trade-off of virtualization is widely discussed in liter-ature. Krylovskiy [10] evaluates Docker for resource con-strained devices, identifying negligible overhead and evenoutperforming native execution in some tests. Similar per-formance characteristics are identified by Felter et al. [11],identifying negligible overhead for CPU and memory per-formance. Felter et al. recommend that for input/output-intensive workloads, virtualization should be used carefully.As Felter et al. use the AUFS storage driver for their evalu-ation, we use the Overlay storage driver as it is consideredto be the default option for Docker in the future.Negligible performance overhead is also identified by Rahoet al. [12] who identify a 3.2% CPU performance decaywhen using Docker on an ARM device. In respect to self-driving vehicles, however, there lacks research using Dockerfor real-time applications. Mao et al. [13] investigate usingDocker for time-sensitive telecommunications networks inthe cloud. A benchmarking tool is used to report the com-putational worst case execution time when executed nativelyand within a container using three different operating systemkernels. The results show that the difference between Dockerand native execution is only 2 µs when using the preempt rtreal-time Linux kernel irrespective to system load. Latencywhen using the real-time kernel is improved by 13.9 timesfor Docker (from 446 µs to 32 µs) and 7.8 times for nativeexecution (from 234 µs to 30 µs) in comparison to a vanillakernel [13]. The authors concluded that in order to satisfyreal-time demands, a finely-tuned real-time Linux kernelmust be used to achieve near native performance.Furthermore, the authors identified considerable overheadwhen using Docker on multiple hosts. This finding is alsoconfirmed by Ruiz et al. [14] who identify a high cost whenusing multiple containers on different nodes. The authorsof [15] also investigate using Docker to realize a modularCPS architecture design. The authors encapsulate computa-tional nodes into containers to improve security and to easesystem administration through modularity and scalability,decoupling the complexity of a CPS into smaller subsystems.They point out the benefit that teams can work independentlyand concurrently on Docker images as well as the need ofusing a real-time enabled Linux kernel. Current literaturepoints to the fact that Docker can be used for CPS due tonegligible overhead identified in current research. However,further evidence is needed for the area of automated driving.

IV. EXPERIMENTS

The aim of the controlled experiment is to systematicallyevaluate the scheduling precision and input/output perfor-

mance of two sample applications; both during native ex-ecution and encapsulated into a sandboxed environment(Docker). Scheduling precision refers to how precisely, inmeasures of time, the CPU scheduler is able to execute anoperation from when the operation was first called. Whereasthe input/output performance refers to measuring the perfor-mance of camera input and disk output, namely the time ittakes capturing an image and saving it on the disk. Through asequence of controlled steps, the sample applications are exe-cuted in four different execution environments. The executionenvironments consist of an alternation of (i) executing thesample applications natively or sandboxed within a Dockercontainer and (ii) executing the sample applications on atarget system with a vanilla or a real-time enabled kernel.Understanding how the respective execution environmentsinfluence the scheduling precision and input/output perfor-mance will ultimately decide how deterministic, with respectto time, the system is to uncover the performance cost ofusing Docker for software deployment.The two sample applications, named Pi and Pi/IO compo-nent, are realized with the open-source development frame-work OpenDaVINCI2. Measurement points in the form oftimestamps are captured during runtime of the sample ap-plications to uncover the timing behavior of the respectiveapplication. The Pi component, used to measure schedulingprecision, is tasked to calculate the next digit of Pi untilit reaches 80% of its designated time slice. The remain-ing 20% should be spent sleeping the process. For theexperiments, the 80% CPU load was established based onour observations when executing real-life scenarios with thetruck. Four measurement points are captured during runtime:the timing of OpenDaVINCI’s middleware (named Overhead1 and Overhead 2), the time duration for calculating Pi(Pi Calculation) and the amount of time the process sleeps(Sleep). The Pi/IO component, used to measure input/outputperformance, is tasked to capture an image (Camera Input)and store it to disk (Disk Output).Treatments used for assessing the impact of factors that arespecific to the execution environment (e.g. execution contextand deployment context). During execution of the sampleapplications, system load is applied to the target systemin order to traverse kernel code paths and to mimic run-time load of a self-driving vehicle. Load is applied to thesystem via a user-space application (stress-ng3), spawningtwo worker threads that apply CPU load at 80% withscheduling priority 48. The controlled experiment is executedon an AEC 6950 embedded personal computer 4. The Linuxkernel version 3.18.25 is chosen for both vanilla and real-time enabled kernel. A usb webcam Logitech c930e is usedfor measuring the I/O performance. Ingo Molnar’s real-timepatch (preempt rt) is used to bring real-time capabilities tothe kernel. The Docker storage driver Overlay is used inthe experiment, instead of the default AUFS driver, which is

2http://www.opendavinci.org3http://goo.gl/o5WuFW4http://www.aaeon.com/en/p /fanless-embedded-computers-aec-6950/

Native [Van]

Native [RT]

Docker[Van]

Docker[RT]

100%

%of

time-

slic

e

A. Scheduling Precision(Closer to 100% is better)

Native [Van]

Native [RT]

Docker[Van]

Docker[RT]

0

2

4

6

8·106

Stan

dard

devi

atio

nin

ns

B. Standard Deviation(Less is better)

Overhead 1 Pi calculationOverhead 2 Sleep

Fig. 2. Scheduling precision results from controlled experiment.

known for performance issues.The results from the controlled experiment give recommen-dation on which execution environment is best suited tomeet real-time requirements. That respective execution envi-ronment is applied to the self-driving heavy vehicle (VolvoFH16 truck), where parts of the controlled experiment areexecuted in order to validate the results. The Pi Component isexecuted exclusively on the self-driving truck as the camerahardware setup is networked and differ greatly from that ofthe controlled experiment and would require modifications tothe sample application, which excludes the Pi/IO componentfor the uncontrolled tests.The intention of the real-life use case is to further under-stand the impact of using Docker for software deploymenton a system with all required operations to enable self-driving capabilities running simultaneously as system load.Consequently, the environment is less-controlled from theresearchers’ access, however, it is more realistic in the senseof operational load. The sample application Pi Componentis used to capture data, with the exact same execution pa-rameters and data collection procedure on precisely the sametarget system, where the difference between the experimentsis seen in the applied system stress.

V. RESULTS

In this section, the results from the experiments are in-troduced. Fig. 2 presents the results of running the firstexperimental unit, namely Pi Component. It is run with afrequency of 100Hz in four execution environments. Bothfigures address the scheduling precision of the executionenvironments. Fig. 2.A depicts the average time deadlinefor each of the four execution environments, while Fig. 2.Bdepicts how deterministic each of the execution environmentsare, i.e. how much does each executing time-slice vary fromthe resulted mean time deadline.Executing the experimental unit on a system with a Linuxvanilla kernel results in an average time deadline violation ofapproximately 10%. Fig. 2.B presents the most deterministicexecution environment is executing the experimental uniton a system with an preempt rt real-time Linux kernel.The standard deviation of the sleep execution on a system

Native [Van]

Native [RT]

Docker[Van]

Docker[RT]

4%

8%

12%

%of

time-

slic

e

A. Input/Output Performance(Less is better)

Native [Van]

Native [RT]

Docker[Van]

Docker[RT]

0

1

2

·107

Stan

dard

devi

atio

nin

ns

B. Standard Deviation(Less is better)

Camera Input Disk Output

Fig. 3. Input/Output performance results from controlled experiment.

Scheduling Precisionη2 Pillai’s Trace P-Value

deployment 0.0001 0.0000 <2.2e-16kernel 0.0678 0.0680 <2.2e-16

deployment:kernel 0.0001 0.0000 <2.2e-16

IO Performanceη2 Pillai’s Trace P-Value

deployment 0.0004 0.0004 <2.2e-16kernel 0.0175 0.0175 <2.2e-16

deployment:kernel 0.0002 0.0002 <2.2e-16

TABLE IIEFFECT SIZE & MANOVA RESULTS

with native execution and Linux vanilla kernel is roughly2400 µs. The total standard deviation of the same executionenvironment is approximately 7000 µs. Both figures showno noticeable difference between executing the experimentalunit in Docker or natively in respect to both the determinismor scheduling precision of a system. Outliers, which areapparent, are included in the results and have not beendisregarded.The two charts in Fig. 3 present the camera and diskperformance for each of the execution environments runningthe second experimental unit, namely Pi/IO Component, at10Hz. Fig. 3.A shows that both operations of capturing animage and saving it to disk consumes on average of approx-imately 7% and 12% of the total time-slice. The executionenvironment with the worst input/output performance is asystem running with a preempt rt real-time Linux kernel andexecuting the experimental unit either natively or in a Dockercontainer. Fig. 3.B presents the standard deviation of theinput and output operations which depicts how deterministiceach execution environment is in regards to its input andoutput performance. Each of the execution environmentshave approximately the same standard deviation. Fig. 3.Bshows that the input and output operations on the systemwith an preempt rt real-time Linux kernel results in a higherstandard deviation, thus less deterministic in regards to suchoperations.A MANOVA test was conducted on all extracted data,including the outliers. This test was conducted to understandthe statistical impact each treatment have on the dependentvariables, i.e. scheduling precision and input and output per-formance. The MANOVA resulted in all treatments having

Native [RT]

Docker[RT]

100%

%of

time-

slic

e

A. Scheduling Precision(Closer to 100% is better)

Native [RT]

Docker[RT]

0

2

4

6

·104

Stan

dard

devi

atio

nin

ns

B. Standard Deviation(Less is better)

Overhead 1 Pi calculationOverhead 2 Sleep

Fig. 4. Scheduling precision results from the uncontrolled experiment.

a significant P-Value thus indicating a significant impactfor all treatments (i.e. alternating deployment and kernel).However, the P-Value can not be fully trusted as this studyhas a vast amount of sample data which carries a risk ofType I error [16], thus an effect size is extracted to fullycomprehend what that significant impact is in reality. Table IIpresents the results from the MANOVA as well as the effectsizes for each of the experimental units. The first threevalues depict the results from each of the alternating factorswithin the execution environment of the experimental unit PiComponent.As the effect size and Pillai’s Trace are both lower for the de-ployment treatment, in comparison with the kernel treatment,it suggests that switching between a vanilla Linux kerneland a preempt rt real-time Linux kernel has a greater impacton the scheduling precision and input/output performance incomparison to the alternation between executing natively andwithin a Docker container.Finally, after executing the controlled experiment, an uncon-trolled experiment was carried out on a machine installedwith the best suitable kernel found through the controlledexperiment, namely a preempt rt real-time Linux kernel.Fig. 4 presents the results from executing the uncontrolledexperiment on the self-driving truck with an applied loadproduced by all software components used for its self-drivingcapabilities. The goal is to understand how Docker impactsthe time critical scheduling precision. Fig. 4.A shows thaton average neither a Docker nor a native execution ofthe experimental unit violated the specified time deadline.Fig. 4.B suggests that a deployment applying native execu-tion performs better compared to executing with a Dockercontainer in regards to the determinism of all executions.However, the standard deviation difference between the twodeployment approaches is a mere 20 µs. The effect size (η2)of the deployment in the uncontrolled experiment is 0.5041which can be considered a medium effect in regards toCohen’s D.All experimental material such as raw data, experimentalunits, and statistical R scripts used for extracting and process-ing the data presented in this section can be found online.5

5https://github.com/docker-rt-research/experimental-material

VI. ANALYSIS & DISCUSSION

It is not sufficient to look at the impact of the deploymentstrategy exclusively to fully comprehend what executionenvironment is suitable to adopt for a system of a self-drivingvehicle. Additional factors play crucial parts of the perfor-mance within such a system. Further factors such as kerneland system load are crucial to acknowledge when decidingupon the execution environment. This is confirmed by theresults, which show that utilizing Docker for the deploymentstrategy of a self-driving vehicle has negligible impact onthe performance of the system. The literature review furtherplays an integral role in the findings, as it has been pointedout in related work that a Docker solution does not addsubstantial overhead to a given system’s performance.The data gathered from all executions in both experimentshave shown that Docker is not the crucial factor to focuson when deciding which execution environment to adoptfor a self-driving vehicle. Both the controlled and uncon-trolled experiments presented similar results in regards tothe scheduling precision, whereas the controlled experimentextended the scope for the input/output performance of thesystem. The data extracted from each environment has shownthat selecting the correct kernel has a greater importance onthe scheduling precision and input/output performance of aself-driving vehicle, where both effect size and the presentedgraphs convey convincing evidence.This is in line with previous research exploring the impact ofDocker utilizing a preempt rt real-time Linux kernel (cf. Maoet al. [13]). They present that the difference between a nativeexecution and executing within a Docker container is a mere2 µs. While the latency is improved by 13.9 times when uti-lizing a preempt rt real-time Linux kernel in comparison to ageneric Linux kernel. Further research has also shown similarresults when utilizing Docker for the deployment strategywhere Felter et al. [11] found that Docker has negligibleoverhead in regards to CPU and memory performance, andKrylovskiy [10] presents negligible overhead introduced byDocker when executed with an ARM CPU.Other performance aspects such as input/output performanceare taken into consideration during the controlled experiment.The results show that executing the application within aDocker container has negligible impact on the input/outputperformance, while utilizing a preempt rt real-time Linuxkernel had a negative impact on the input/output performanceof the application. This may be explained by the system’spreemptive approach where the input/output operations canbe preempted by other processes and thus, increasing thetime required to execute the operations.The results regarding the determinism of the executionenvironment captured in the uncontrolled experiment on theself-driving truck differ from those captured in the controlledexperiment. Where the uncontrolled experiment shows thatDocker has an impact on how deterministic a system iswhen executed alongside components, which enable the self-driving capabilities of the truck. However, the results suggestthat the load introduced in the controlled experiment is no-

ticeably more exhausting in relation to the truck experimentas its highest standard deviation is around 7000 µs while thetruck’s highest standard deviation is around 60 µs.

A. Threats to Validity

The four perspective of validity threats presented by Runesonand Host [17] are discussed. In respect to construct validity,the sample applications are realized with OpenDaVINCI toensure a high degree of software correctness and complete-ness, meeting the design requirements for real-time systems.For internal validity, a number of strategies are used tolimit the risk of an unknown factor impacting the datacollected. Namely, the execution of the sample applicationsis carried out by a script to ensure precise reproducibility,all peripherals such as networking are detached and data iscollected via serial communication to limit additional loadto the system. In respect to external validity, the results ofthis study can be applied to time-sensitive applications inrespect to the hardware and software used. The hardwareused in this study is industrial grade, making the experimentreproducible and the results relatable to similar contexts. Forconclusion validity, there exists a possibility of Type I andType II statistical errors. Due to the sample size of the datacollected, Type I and Type II errors are considerable sincewhere an increasing sample size will result in a decreasingP-value. For that reason, this study has put less emphasis onthe P-value, taking a larger consideration on the effect sizewhen evaluating the data.

VII. CONCLUSION & FUTURE WORK

A literature review and two experiments, one controlled andone uncontrolled, have been carried out to fully comprehendhow an alternation of factors within the execution environ-ment influence the execution performance of a system fora self-driving vehicle. More specifically, these experimentssought to uncover which Linux kernel is most suitable forsuch a context, and whether or not utilizing Docker as atechnical platform for a software deployment strategy has animpact on the time critical deadlines specified for the real-time application.Initially, the controlled experiment intended to identify themost appropriate kernel in terms of scheduling performance.The most appropriate kernel was later implemented into anuncontrolled environment of a self-driving truck, which par-ticipated in the 2016 Grand Cooperative Driving Challenge(GCDC). The research goal was to identify whether Dockeris a suitable technical environment to realize continuousintegration, continuous deployment, and continuous exper-imentation on the self-driving truck. Our results show thatDocker is not the critical factor to consider when selectingan execution environment; however, the Linux kernel in usewas identified as having a greater impact on the schedulingprecision and input/output performance of the software.Future research is needed to understand how Docker be-haves with respect to network performance, when severalcomponents are executed from within Docker containers andthere is communication between separate computer nodes.

Moreover, on-road tests will be considered in future workin order to cover variables such as CPU load and memoryfootprint under real-life conditions.

REFERENCES

[1] T. Savor, M. Douglas, M. Gentili, L. Williams, K. Beck, andM. Stumm, “Continuous deployment at facebook and oanda,” inProceedings of the 38th International Conference on SoftwareEngineering Companion, ser. ICSE ’16. New York, NY, USA:ACM, 2016, pp. 21–30. [Online]. Available: http://doi.acm.org/10.1145/2889160.2889223

[2] What is docker? [Online]. Available: http://www.docker.com/what-docker

[3] D. Merkel, “Docker: lightweight Linux containers for consistentdevelopment and deployment,” Linux Journal, no. 239, Mar. 2014.[Online]. Available: http://dl.acm.org/citation.cfm?id=2600241

[4] Let me contain that for you (lmctfy). [Online]. Available:https://github.com/google/lmctfy

[5] P. Masek and M. Thulin, “Container based virtualisation for softwaredeployment in self-driving vehicles,” Master Thesis, Chalmers Uni-versity of Technology, 2016.

[6] C. Wohlin, “Guidelines for snowballing in systematic literaturestudies and a replication in software engineering,” in Proceedingsof the 18th International Conference on Evaluation and Assessmentin Software Engineering, ser. EASE ’14. New York, NY,USA: ACM, 2014, pp. 38:1–38:10. [Online]. Available: http://doi.acm.org/10.1145/2601248.2601268

[7] M. Wood, J. Daly, J. Miller, and M. Roper, “Multi-method research:An empirical investigation of object-oriented technology,” Journalof Systems and Software, vol. 48, no. 1, pp. 13 – 26, 1999.[Online]. Available: http://www.sciencedirect.com/science/article/pii/S0164121299000424

[8] Scopus digital library. [Online]. Available: http://www.scopus.com[9] C. Berger, “Testing continuous deployment with lightweight multi-

platform throw-away containers,” 41th Conference on Software Engi-neering and Advanced Applications (SEAA), 2015.

[10] A. Krylovskiy, “Internet of things gateways meet linux containers:Performance evaluation and discussion,” in Internet of Things (WF-IoT), 2015 IEEE 2nd World Forum on, Dec 2015, pp. 222–227.

[11] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio, “An updatedperformance comparison of virtual machines and linux containers,” inPerformance Analysis of Systems and Software (ISPASS), 2015 IEEEInternational Symposium on, March 2015, pp. 171–172.

[12] M. Raho, A. Spyridakis, M. Paolino, and D. Raho, “Kvm, xenand docker: A performance analysis for arm based nfv and cloudcomputing,” in Information, Electronic and Electrical Engineering(AIEEE), 2015 IEEE 3rd Workshop on Advances in, Nov 2015, pp.1–8.

[13] C. N. Mao, M. H. Huang, S. Padhy, S. T. Wang, W. C. Chung,Y. C. Chung, and C. H. Hsu, “Minimizing latency of real-timecontainer cloud for software radio access networks,” in 2015 IEEE7th International Conference on Cloud Computing Technology andScience (CloudCom), Nov 2015, pp. 611–616.

[14] C. Ruiz, E. Jeanvoine, and L. Nussbaum, “Performance evaluationof containers for hpc,” in Euro-Par 2015: Parallel ProcessingWorkshops: Euro-Par 2015 International Workshops, Vienna, Austria,August 24-25, 2015, Revised Selected Papers. Cham: SpringerInternational Publishing, 2015, pp. 813–824. [Online]. Available:http://dx.doi.org/10.1007/978-3-319-27308-2 65

[15] P. Gonzalez-Nalda, I. Etxeberria-Agiriano, I. Calvo, and M. C.Otero, “A modular cps architecture design based on ros and docker,”International Journal on Interactive Design and Manufacturing(IJIDeM), pp. 1–7, 2016. [Online]. Available: http://dx.doi.org/10.1007/s12008-016-0313-8

[16] V. B. Kampenes, T. Dyba, J. E. Hannay, and D. I. Sjøberg, “Asystematic review of effect size in software engineering experiments,”Information and Software Technology, vol. 49, no. 11, pp. 1073–1086,2007.

[17] P. Runeson and M. Host, “Guidelines for conducting and reportingcase study research in software engineering,” Empirical Softw.Engg., vol. 14, no. 2, pp. 131–164, Apr 2009. [Online]. Available:http://dx.doi.org/10.1007/s10664-008-9102-8