a placement architecture for a container as a service ... · research open access a placement...

RESEARCH Open Access

A placement architecture for a container asa service (CaaS) in a cloud environmentMohamed K. Hussein1* , Mohamed H. Mousa1,2 and Mohamed A. Alqarni2

Abstract

Unlike a traditional virtual machine (VM), a container is an emerging lightweight virtualization technology thatoperates at the operating system level to encapsulate a task and its library dependencies for execution. TheContainer as a Service (CaaS) strategy is gaining in popularity and is likely to become a prominent type of cloudservice model. Placing container instances on virtual machine instances is a classical scheduling problem. Previousresearch has focused separately on either virtual machine placement on physical machines (PMs) or container, oronly tasks without containerization, placement on virtual machines. However, this approach leads to underutilizedor overutilized PMs as well as underutilized or overutilized VMs. Thus, there is a growing research interest indeveloping a container placement algorithm that considers the utilization of both instantiated VMs and used PMssimultaneously.The goal of this study is to improve resource utilization, in terms of number of CPU cores and memory size forboth VMs and PMs, and to minimize the number of instantiated VMs and active PMs in a cloud environment. Theproposed placement architecture employs scheduling heuristics, namely, Best Fit (BF) and Max Fit (MF), based on afitness function that simultaneously evaluates the remaining resource waste of both PMs and VMs. In addition,another meta-heuristic placement algorithm is proposed that uses Ant Colony Optimization based on Best Fit(ACO-BF) with the proposed fitness function. Experimental results show that the proposed ACO-BF placementalgorithm outperforms the BF and MF heuristics and maintains significant improvement of the resource utilizationof both VMs and PMs.

Keywords: Cloud container, Virtual machine placement, Container placement, Two-tier placement algorithm, Antcolony, Best fit, Max fit

IntroductionCloud computing is the on-demand delivery of comput-ing applications, storage and infrastructure as servicesprovided to customers over the Internet. Users pay onlyfor the services that they use according to a utility char-ging model [1–3]. Cloud services are typically providedin three different service models: (1) Infrastructure as aService (IaaS), which is a computing infrastructure inwhich storage and computing hardware are providedthrough API and web portals; (2) Platform as a Service(PaaS), which is a platform for application developmentprovided as services, such as scripting environments anddatabase services. (3) Software as a Service (SaaS), whichprovides applications as services, (e.g. Google services).

Virtual machine (VM) technology is the leading execu-tion environment used in cloud computing. Theresources of a physical machine (PM), such as CPUs,memory, I/O, and bandwidth, are partitioned amongindependent virtual machines, each of which runs an in-dependent operating system. A Virtual Machine Monitor(VMM), also called a hypervisor, manages and controlsall instantiated VMs, as shown in Fig. 1a.A container is a new lightweight virtualization tech-

nology that does not require a VMM. A containerprovides an isolated virtual environment at the oper-ating system level. Furthermore, different containersexecuting on a specific operating system share thesame operating system kernel; consequently, executinga container does not require launching an operatingsystem, as shown in Fig. 1b.

© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made.

* Correspondence: [email protected] of Computers and Informatics, Suez Canal University, Ismailia, EgyptFull list of author information is available at the end of the article

Journal of Cloud Computing:Advances, Systems and Applications

Hussein et al. Journal of Cloud Computing: Advances, Systems and Applications (2019) 8:7 https://doi.org/10.1186/s13677-019-0131-1

http://crossmark.crossref.org/dialog/?doi=10.1186/s13677-019-0131-1&domain=pdf

http://orcid.org/0000-0002-4635-4969

http://creativecommons.org/licenses/by/4.0/

mailto:[email protected]

To provide containers as services, a relatively newcloud service model based on container virtualization,called Container as a Service (CaaS), has emerged. CaaSsolves the problem of applications developed in a certainPaaS environment and whose execution is restricted tothat PaaS environment’s specifications. Therefore, CaaSfrees the application, making it completely independentof the PaaS specification environment, and eliminatesdependencies. Hence, container-based applications areportable and can execute in any execution environment[4]. Several container services are currently available onpublic clouds, including Google Container Engine andAmazon EC2 Container Service (ECS).Docker, which is an open platform for executing

application containers [5], comprises an ecosystem forcontainer-based tools. A Docker Swarm functions as ascheduler for the containers on VMs. However, theDocker Swarm placement algorithm simply mapscontainers onto the available resources in round-robinfashion [6], and the round-robin placement algorithmdoes not consider resource usage or the utilization ofeither PMs or VMs. The problem of container or taskplacement has been widely studied [7–9]. Thisproblem is critical because it has a direct impact onthe utilization of PMs, the performance of VMs, thenumber of active PMs, the number of instantiatedVMs, energy consumption, and billing costs. Thispaper investigates the following research problem:

� Container placement: This involves mapping newcontainers to VMs by considering VMs placementson the PMs at the same time. The placement aimsto minimize instantiated VMs, minimize active PMsand to optimize the utilization of both the activePMs and VMs in terms of number CPU cores andmemory usage.

In this paper, we propose a placement architecturefor containers on VMs. The proposed architecture aimsto maximize the utilization of both VMs and PMs andto minimize the number of instantiated VMs and active

PMs. To this end, we explore the Best Fit and Max Fitheuristics along with a fitness function that evaluatesresource usage as scheduling algorithms. Furthermore,a meta-heuristic Ant Colony Optimization based onBest Fit (ACO-BF) in combination with the proposedfitness function is also proposed.The scope of the study is as follows:

� Each job consists of a set of independent tasks, andeach task executes inside a container.

� Each task is represented by a container whoserequired computing resources, CPU and memory, areknown in advance. The CPU and memory resourcesare the only resources considered in the study.

� Each container executes inside a single VM.

The remainder of this paper is organized as follows.Section 2 discusses the related research background.The proposed placement architecture and algorithms areintroduced Section 3. Section 4 discusses the experimen-tal evaluation and presents the feasibility study of theproposed algorithms. Section 5 concludes the paper andpresents future work.

Background and related workCloud computing is led by the virtualization technology. Avirtual machine is an emulation of physical hardware re-sources that runs an independent operating system on topof a VMM, also called a hypervisor. The hypervisor acts asa management layer for the created virtual machines. Re-cently, however, a new virtualization technology hasemerged, namely, containerization. A container is a light-weight virtual environment that shares the operating sys-tem kernel with other containers. Docker is an open-sourceapplication development platform in which developed ap-plications can execute in either a virtual machine or on aphysical machine [6]. Docker Containerization is consid-ered lightweight for two main reasons: (1) it significantlydecreases container startup times because the containersdo not require launching a VM or an operating system,

Physical machines (PMs)

Host Operating System

ContainerE

ngine

APPC1

AppCn

Virtual machines (VMs)

Physical machines

a b

(PMs)

Host Operating System

VM1OS1

VM2OS2

VMnOSn

Hypervisor

Fig. 1 Virtualization technology. a. Virtual machine architecture. b. Container virtualization architecture

Hussein et al. Journal of Cloud Computing: Advances, Systems and Applications (2019) 8:7 of 15

and (2) communications are performed through systemcalls using the shared operating system kernel—unlike theheavyweight communication performed by a virtual ma-chine hypervisor.To this end, a container is considered an isolated

workload unburdened by the overhead of a hypervisor.An application’s tasks execute inside the containers, andthe containers are scheduled to virtual machines, whichare mapped to physical machines. Consequently, it ischallenging to design an effective placement algorithmthat considers the utilization of both VMs and PMs.Placement problems have been considered in the

literature from the following different perspectives:

� Placement of virtual machines on physical machines� Placement of containers on physical machines� Placement of containers on virtual machines� Placement of containers and virtual machines on

physical machines.

Placement of VMS on PMs (VM-PM)A virtual machine is a simulation of physical hard-ware resources, including processors, memory, and I/O devices. Each VM runs an independent operatingsystem on top of the VMM. Previous studies on theproblem of VM-PM placement were mainly intendedto optimize the utilization of PMs in IaaS cloud infra-structure and to maintain certain Quality of Service(QoS) requirements of the applications submitted toIaaS clouds. Much of this research was conducted toinvestigate VM placement and provisioning on PMs,including static and dynamic scheduling, and optimalPM utilization was their main objective [9–11].

Placement of containers on PMs (C-PM)Docker is an operating system-level virtualization mech-anism that allows Linux containers to execute independ-ently on any host in an isolated environment. Forcontainer placement, algorithms such as Docker Swarmand Google Kubernets [12, 13] have been proposed. TheDocker Swarm allocation algorithm simply maps con-tainers to the available resources in round-robin fashion[6]; however, many other scheduling algorithms have beenproposed. The main goals of these scheduling algorithmsare to either maximize the utilization of PMs or tomaximize container performance. For example, in [14], anarchitecture called OptiCA was proposed in which eachcomponent of the distributed application is executed in-side a container, and the containers are deployed on PMs.The main goal is to optimize the performance of big-dataapplications within the constraints of the available coresof the PMs. In [15], an algorithm was proposed tominimize the energy consumption of a data center by

placing applications on the PMs efficiently under a certaindefined service level agreement constraint while simultan-eously minimizing the number of required PMs. A simula-tion was used to model the workload using a Poissondistribution with a queuing routing strategy to map theworkload to the determined PMs.

Placement of containers on VMs (C-VM)The placement algorithms of VMs on PMs are mainlyintended to optimize PMs utilization. Currently,containerization technology has been used to optimize theutilization of VMs. Application containers are provided byAmazon EC2 in the form of a CaaS cloud service. In theCaaS model, tasks execute inside containers that are allo-cated to VMs to minimize the application response timeand optimize the utilization of the VMs. For example, theproposed queuing algorithm in [15] can be used to mapthe containers directly onto VMs rather than onto PMs tominimize the required number of VMs and, consequently,to minimize energy consumption. In [16], a placement al-gorithm for containers on VMs was proposed. The maingoal of the proposed placement algorithm was to reducethe number of instantiated VMs to reduce billing costsand power consumption. The proposed model used theconstraint satisfaction programming (CSP) approach. In[17], a meta-heuristic placement algorithm was proposedthat dynamically schedules containers on VMs based onan objective function that includes the number of PMs,PM utilization, SLA violations, migrations and energyconsumption. In [18], the proposed placement algorithmwas based on using Constraint Programing (CP) to mapcontainers to VMs with the goal of minimizing the num-ber of required VMs.

Two-tier placement of containers and VMs on PMs (C-VM-PM)The two-tier placement algorithm schedules containersand VMs on PMs. The objectives of such placementalgorithms are to (1) maximize the utilization of PMsand VMs, (2) minimize the number of PMs and VMs,and (3) improve the overall performance of the executedapplications. This type of placement algorithm is themain concern of this research.For example, in [19], a two-level placement algo-

rithm is proposed for scheduling tasks and VMs onPMs simultaneously. The main goal of the proposedalgorithm is to maximize PMs utilization. The place-ment algorithm first schedules VMs on the PMs usingthe Best Fit algorithm and then schedules tasks onthe VMs using Best Fit. The results are comparedwith the those of FCFS scheduler. However, the pro-posed scheduler focuses on PMs utilization and doesnot simultaneously consider the utilization of bothVMs and active PMs.


In [20], the Best Fit algorithm is used to load-balance thetasks over VMs by first considering the placement of VMson active PMs using the Best Fit algorithm. This approachaims to ensure moderate PMs utilization. However, thefocus of the study involves VM placement on PMs.In [21], a placement algorithm is proposed based on a

fitness function that considers the utilization of both thePMs and the VMs. The Best Fit placement algorithm isused along with the fitness function to evaluate the selec-tions of both PMs and VMs. The experimental resultsshow that the proposed algorithm improved PMutilization compared with the Best Fit algorithm withoutthe fitness function. In this paper, a similar fitness functionis proposed based on the remaining available resourcepercentages of both VMs and PMs. The fitness function isemployed in the placement algorithms. Furthermore, inthis paper, we explore using different heuristics, such asBest Fit, Max Fit and the meta-heuristic Ant ColonyOptimization (ACO) algorithm.

The proposed (C-VM-PM) placement architecture andscheduling algorithmsAs mentioned above, most existing research has focusedon the problem of scheduling containers on either VMs orPMs or on the placement of VMs on PMs. Thus, the exist-ing provision architectures and container-scheduling algo-rithms for VMs do not consider the VMs and PMattributes together. This leads to major fragmentation ofthe PM’s resources and poor PMs utilization. For example,consider the placement scenario for container C5 usingthe configuration shown in Fig. 2. There are several place-ment options, and each placement decision impacts theutilization of VMs and PMs. The first placement option isthe placement of container C5 in either VM1 or VM2.This placement improves the utilization of VM1 or VM2.

The second placement option is the placement of con-tainer C5 in VM4. This placement option improves theutilization of PM2 as well as the utilizations of VMs onthat PM. Finally, the last placement option is the place-ment of container C5 in VM4. This placement option isthe worst option where an unnecessary PM is switchedon. Without this placement option, PM3 can be put tosleep to minimize the number of PMs used.This section presents the proposed placement archi-

tecture and scheduling algorithms proposed to improvePM and VM utilization while scheduling containers in aCaaS cloud service model. The architecture aims tominimize the number of instantiated VMs and activePMs and maximize their resource utilization.

The proposed placement architectureIn CaaS, tasks execute inside containers that are sched-uled on VMs, which, in turn, are hosted on PMs. The ob-jective of the placement architecture is to improve theoverall resources utilization of both VMs and PMs, and tominimize the number of instantiated VMs and the num-ber of active PMs. An overall view of the proposed archi-tecture is shown in Fig. 3. The architecture consists of thefollowing components:

Physical machine controller (PMC)The PMC is a daemon that runs on each PM. ThePMC collects information such as the utilization ofeach PM and its resource capacity. The PMC sendsthe collected data to the container scheduler.

VM controller (VMC)A VMC is a daemon that runs on each instantiated VM.The VMC collects information such as the resource cap-acity of the VMs and its used resources. The VMC sends

Fig. 2 An example of a container placement


the collected data to the container scheduler and to theVM information repository.

VM information repository (VMIR)The VMIR maintains information about the types ofVMs that can be instantiated and information abouteach instantiated VM.

VM provisioning (VMP)The VMP schedules the placement of newly instantiatedVMs on available PMs using the information availablefrom the VMIR and PMC. Furthermore, the VMP is re-sponsible for VMs provisioning.

Container scheduler (CS)The CS determines the placements of containers on theVMs. The placement decisions are based on the infor-mation available from the PMC and the VMIR.In the proposed architecture, each job consists of a

number of independent tasks, each of which executes in-side a container. The Container Scheduler (CS) contactsthe VMIR to obtain the available resource capacity ofeach VM. The CS applies the placement algorithms thatare proposed in the following subsections to find aplacement for each container on a VM. When the CSfinds a placement, it contacts the VMC to launch thecontainer. When there is no VM available with enoughphysical resources to host the container, a new VM must

Fig. 4 Sequence diagram for scheduling a container in the proposed architecture

PMs

VMsInformation

A job consists of a number of containers requests placement VMs

Controller

PM1PM2

VMM

VM1 VM2 VMm

ContainersCi,k

PMn

PMscontroller

VMsProvisioning

ContainerScheduling

Fig. 3 The proposed architecture


be instantiated—either on a new or currently active PM.To accomplish this task, the CS asks the VMP to instan-tiate a new VM. The VMP finds a placement for the newVM using the information provided by the VMIR andthe PMC. The VMP instantiates the new VM and up-dates the VMIR. Finally, the VMC starts the containeron the newly instantiated VM. Figure 4 shows the se-quence diagram for container placement in the proposedarchitecture.The following subsection presents a mathematical for-

mulation for the problem of container placement onVMs and PMs. The main objective is to improve theoverall utilization of VMs and PMs. Based on the pro-posed formulation, heuristic and meta-heuristic schedul-ing algorithms will be presented.

Problem formulationThis subsection presents the mathematical formulationof the objective function for the proposed architecture,

which minimizes the number of active VMs and PMsand maximizes the utilization of both VMs and PMs.Table 1 summarizes the symbols used in this section.The goal of the objective function is to minimize the

number of physical machines:

minimize Npm ¼Xj

Nvmpm j

We consider the following constraints on the schedul-ing decisions:

C k;i; jð Þ ¼ f xð Þ ¼ 1; container k is scheduled to vmi; j

0; otherwise

�ð1Þ

vmi j ¼ f 1; vmi is scheduled on pmj

0; otherwise ð2ÞXj

Xi

C k;i; jð Þ ¼ 1∀k ð3Þ

Xi

vmC i; j;rð Þ≤SC j;rð Þ∀r

¼ cpu cores;memory; bandwidth; storagef g; jð4ÞX

i

Xk

CC k;i; j;rð Þ≤SC j;rð Þ∀resource r; j: ð5Þ

Equation (4) ensures that the sum of each resource typeallocated by all VMs on each PM is less than or equal tothe total amount of that resource available on the PM.Equation (5) ensures that the sum of each resource re-

quested by the containers on all VMs vmij on the physicalmachine pmj is less than or equal the total capacity SC(j, r)

of that resource on that physical machine pmj. This condi-tion applies to computational resources, such as CPUcores, memory, storage, and bandwidth. The placementalgorithms should maximize the resource utilization on

Fig. 5 The Best Fit algorithm for container-VM-PM placement

Table 1 Symbol and descriptions

Symbol Description

vmij Virtual machine i on physical machine j.

pmj Physical machine j.

C(k, i, j) Container number k on vmij

CC(k, i, j, r) The container k capacity of resource r on vmij

vmC(i, j, r) The resource r capacity of vmij

SC(j, r) The resource r capacity of pmj

NCvmi; j Number of containers on vmij

Nvmpm jNumber of VMs on pmj

Npm Total number of PMs

CPUrequired The required number of CPU cores for new containerplacement

memoryrequired The required amount of memory for new containerplacement


the VMs and PMs. This can be achieved by minimizingthe resource waste on both the VMs and PMs simultan-eously. Equation (6) calculates the normalized resourcewaste on a PM considering only the number of CPU coresand the memory resources.

PF jð Þ ¼ w1 �PM availableCPU cores

j

SC j;r¼cpu coresð Þþ w2

�PM availablememoryj

SC j;r¼memoryð Þ

ð6Þ

w1 and w2 are weights constants. PM availableCPU coresj

and PM availablememoryj represent the available number

of cores and the amount of memory on pmj after

assigning the requested physical resources, respectively,and they are calculated using Eqs. (7, 8).

PM availableCPUcoresj ¼ SC j;r¼cpu coresð Þ

−ðXi

vmij �Xk

CC k;i; j;r¼cpu coresð Þ

!

þCPUrequiredÞ

ð7Þ

PM availablememoryj ¼ SC j;r¼memoryð Þ

−ðXi

vmij �Xk

CC k;i; j;r¼memoryð Þ

!

þmemoryrequiredÞ

ð8Þ

Fig. 7 The ACO algorithm for container-VM-PM placement

Fig. 6 The Max Fit algorithm for container-VM-PM placement


VF i; jð Þ ¼ w1 �VM availableCPU cores

ij

vmC i; j;r¼CPU coresð Þþ w2

�VM availablememoryij

vmC i; j;r¼memoryð Þþ PF jð Þ � vmij

ð9Þ

Equation (9) calculates the normalized resource wasteof a VM while considering the resource waste of the PMhosting that VM. VM availableCPU cores

ij and VM

availablememoryij represent the number of cores available

and the amount of memory on vmij after assigning therequested virtual resources, respectively, and they arecalculated using Eqs. (10) and (11).

VM availableCPUcoresij ¼ vmC i; j;r¼CPU coresð Þ

−ðXk

C k;i; jð Þ � CC k;i; j;r¼CPU coresð Þ� �

þCPUrequiredÞ

ð10Þ

VM availablememoryij ¼ vmC i; j;r¼memoryð Þ

−ðXk

C k;i; jð Þ � CC k;i; j;r¼memoryð Þ� �

þmemoryrequiredÞð11Þ

The placement algorithm should make its decisionsbased on the minimum values of Eqs. (6) and (9).

Fig. 8 A sample of the vCore and memory requests for 1000 tasks

Fig. 9 The number of tasks to be scheduled during a 1000-s period


However, this problem is NP-hard and the difficulty in-creases exponentially as the number of containers in-creases [9]. The following subsection discusses theheuristic and meta-heuristic scheduling algorithms.

The proposed scheduling algorithmsThis subsection discusses the proposed scheduling al-gorithms for scheduling containers on VMs andscheduling VMs on PMs. The problem can be viewedas a multitier bin-backing problem in which the VMsare considered the bins for the containers, while sim-ultaneously, the PMs are considered the bins for theVMs. The goals of the proposed scheduling algo-rithms are to (1) maximize PMs utilization and (2)maximize VMs utilization. To achieve these goals, wepropose three algorithms using the mathematical for-mulation discussed in Section 3.2. The algorithms areBest Fit, Max Fit, and a meta-heuristic based on AntColony Optimization and Best Fit.

Best fit algorithmThe Best Fit algorithm (BF) is a heuristic that is usedto optimize the bin packing problem. The BF algo-rithm is widely used in the literature to find place-ments for VMs on PMs [22]. In this study, the BFalgorithm is used to find a placement for containerson VMs by considering the resource utilizations ofboth the VMs and the PMs. The BF algorithm pre-sented in Fig. 5 schedules containers onto VMs thathave the least available resource capacity that can ac-commodate the resources required by the new con-tainer. This is achieved by scheduling the containerto the VM with the minimum resource wastage, usingEq. (9). When no existing VM has sufficient availableresources to accommodate the resources required bythe new container, a new VM must be instantiated.The same idea is applied to the placement of VMs

onto PMs: a new VM is instantiated on the PM with theleast available resource capacity that can accommodatethe resources required by the new VM using Eq. (6). Fi-nally, when there is no PM available whose available re-source capacity fits the resources required by the newVM, a new PM is activated. The Best Fit algorithm isshown in Fig. 5.

Max fit algorithmThe Max Fit (MF) algorithm is similar to the Best Fitheuristic. However, the main difference is that MaxFit elects to place containers on VMs that have themaximum available resource capacity and can accom-modate the resources required by the new container.Thus, in Max Fit, containers are scheduled to VMsthat have the maximum resource wastage using Eq.(9). When no VM has sufficient available resources to

accommodate the resources required by the new con-tainer, a new VM must be instantiated. The new VMis instantiated on the active PM with the maximumavailable resource capacity that can accommodate theresources required by the new VM using Eq. (6). Fi-nally, when no PM exists whose available resourcecapacity can accommodate the resources required bythe new VM, a new PM is activated. The Max Fit al-gorithm is shown in Fig. 6.

Ant colony optimization (ACO)ACO is a meta-heuristic inspired by real ant coloniesand their collective foraging behavior for food sources.ACO is a probabilistic method used to solve discreteoptimization problems, such as finding the shortest pathin a graph [23]. In the search for food, an ant leaves thenest and travels along a random path to explore the sur-rounding area for food. During the search, the ant pro-duces pheromone trails along its path. Other ants smellthe pheromones and choose the path with a higherpheromone concentration. During the ant’s return trip,it releases a quantity of pheromone trails proportional tothe quantity and quality of the food it found. All theremaining ants of the colony will follow paths withhigher pheromone concentrations at a higher probabilitythan they follow paths with lower pheromone concentra-tions. Eventually, all the ants will follow a single optimalpath for food. For m ants and n possible paths, each antdetermines its path according to the concentration ofpheromone trail in each path of the possible paths. Ini-tially, all the ants select a path randomly because of thenegligible differences in pheromone quantity among thepaths [24, 25].

Table 2 Virtual machine types of instances

VM Type vCPU Memory Size (GB) VM Type vCPU Memory Size (GB)

Type 1 1 1 Type 9 16 64

Type 2 1 2 Type 10 16 128

Type 3 2 4 Type 11 32 128

Type 4 2 8 Type 12 32 256

Type 5 4 8 Type 13 48 256

Type 6 4 16 Type 14 48 256

Type 7 8 32 Type 15 64 1024

Type 8 8 64 Type 16 64 2048

Table 3 Physical machine configuration attributes

Server Type Number CPUs Number of Cores/CPU Memory Size (GB)

Type 1 32 4 1024

Type 2 64 8 2048


The ACO algorithm has proven effective as anoptimization algorithm for a number of problems, in-cluding the traveling salesman and job shop schedul-ing [26] problems. Furthermore, the ACO algorithmhas been used to efficiently schedule VMs on cloudresources with the objective of minimizing total re-source wastage and power consumption [27]. TheACO has also been used to schedule tasks on cloudVMs with the goal of load-balancing the tasks on theVMs and reducing the make span of the tasks [28]. Anumber of schedulers have adopted the ACO algo-rithm to load balance tasks on cloud resources [29,30]. The following proposed algorithm uses ACO tofind container placements on VMs with the objectiveof maximizing the utilization of VMs and PMs whileminimizing the number of instantiated VMs and ac-tive PMs.In a search for container placement on vmij and pmj

with the objective of maximizing the used PMs and VMsutilization, the following equation represents the phero-mone trail of that container placement for an ant k initeration (t + 1).

τkij t þ 1ð Þ ¼ 1−ρð Þτij tð Þ þXmk¼1

Δτkij tð Þ; ð12Þ

where Δτkij represents the amount of pheromone releasedby an ant due to the quality of the placement of a con-tainer on vmij and pmj. Here, ρ is a constant that repre-sents the rate of pheromone evaporation to simulate theeffect of the evaporation of the pheromone at each step,and Δτkij is calculated using the following equation:

Δτkij ¼1

VF i; jð Þ ;VF i; jð Þ is calculated using Equation 9ð Þ

ð13ÞVF(i, j) is the fitness of the placement on vmij and rep-

resents the probability that an ant k will select the place-ment of a container on vmij and pmj.

Pkij tð Þ ¼

τij tð Þ� �αηij tð Þ� �β

Xs

τis tð Þð Þα ηis tð Þ� �β ; α; β are heuristic constants;

ð14Þwhere ηij(t) is a heuristic function that represents thequality of a container placement, and ηis(t) is calculatedas follows:

ηij tð Þ ¼Q

VF i; jð Þ ;Q is a heuristic constant; ð15Þ

The complete algorithm is shown in Fig. 7. The pro-posed algorithm starts by calculating the total amountof resources available in the instantiated set vmij. If theavailable resources are below the total amount ofresources required by the set of containers C, a newsubset of VMs is instantiated using the BF algorithmthat satisfies Eqs. (6) and (9), which maximize the VMsand PMs utilization. The algorithm starts by calculating

Fig. 10 Number of instantiated VMs

Table 4 Parameter settings of the conducted experiments

Parameter Description Value

nContainer Number of containers request scheduling 25–250

Max_Iteration Maximum number of iterations in ACO 100

nAnts Number of ants 8

α Heuristic constant 0.1

β Heuristic constant 2

Q Heuristic constant 1

ρ Evaporation rate 0.2

w1 First resource normalization weight 0.5

w2 second resource normalization weight 0.5


the initial pheromone trail matrix τij and the place-ment cost matrix ηij using Eqs. (13) and (15), respect-ively. During each iteration, each ant antk finds aplacement for the set of containers Ck on vmij with aprobability calculated using Eq. (14). The selection ofvmij is performed using the roulette wheel methodwhere a random number ϵ [0, 1] is generated and acumulative sum of the probabilities is calculated. Thecumulative sum probabilities are arranged in ascend-ing order, and the cumulative probability equal to orgreater than the generated random number is se-lected. After each iteration, the ACO algorithm up-dates the pheromone trail matrix τij. The algorithmiterates until the maximum number of iterations isreached or the best placement of the containers onthe subset of VMs is found.

Performance evaluationThis section presents the experiments conducted toevaluate the proposed placement architecture andscheduling algorithms. A program is designed anddeveloped using MATLAB to simulate the proposedplacement algorithms. A simulation-based evaluationis adopted because it allows the environment param-eters to be controlled and the experiments to berepeated under different constraints. The experi-ments are conducted to evaluate the proposedscheduling algorithms in terms of the utilization ofVMs and PMs. The following subsections discuss theworkload utilized for the simulation, the parametersettings and the evaluation of the proposed algo-rithms for effective placement of container-basedcloud environments.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

25 50 75 100 125 150 175 200 225 250

sM

Vfo

noitazilituU

PC

Number of Containers

Max Fit Best Fit ACO

Fig. 11 CPU utilization of instantiated VMs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

25 50 75 100 125 150 175 200 225 250

sM

V fo noitazilitu yrome

M



Fig. 12 Memory utilization of instantiated VMs


Google cloud workloadA workload is a set of jobs submitted for allocation andexecution in a cloud environment. The workload datasetused in the evaluation is the Google Cloud trace log,which was published in 2011 [31]. The Google realworkload dataset is used because it is widely used in theliterature for the purpose of workload placement andscheduling evaluations. In this paper, the workload isused to evaluate the applicability of the proposed place-ment algorithms in real world scenarios. The workloadconsists of a number of tables that describe the submit-ted task requirements, machine configurations, and taskresource usage. The Task Event table contains informa-tion such as time, job ID, and task ID. Additionally, thetable includes normalized data, such as resource re-quests for CPU cores and RAM. The number of vCoresused to denormalize the CPU requests in the task Eventtable is 64. The amount of memory used to denormalizememory requests in the task Event table is 1024 GB. Fig-ure 8 shows a sample of the vCore and memory requestsfor 1000 tasks in the Google cloud workload dataset,

while Fig. 9 shows the number of tasks received forscheduling during a 1000-s period in the sample Googleworkload. Each task in the workload, the task Eventtable, is used to represent a container that requires aplacement using the available VMs and PMs configura-tions described in the following subsections. The re-quired CPU cores and memory request for each task,after denormalization, are considered the required re-sources for container placement.

Virtual machine configurationThe VM type repository includes several types of VMsthat can be instantiated. Each VM type has distinct re-source characteristics, including the number of vCPUs,the memory size, and the storage size. Each vCPU in aVM is allocated one core of the PM. Table 2 presentsthe different VM types used in the experiment.

PM configurationTable 3 shows simulated configurations of the physicalmachines used in the evaluation experiment.

0

2

4

6

8

10

12

14

16

25 50 75 100 125 150 175 200 225 250

sM

Pdetavitca

foreb

muN



Fig. 13 Number of activated PMs

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

25 50 75 100 125 150 175 200 225 250

sM

P detavitca fo noitazilitu U

PC



Fig. 14 CPU utilization of activated PMs


Parameters settingThe simulation was run on a PC with an Intel Core i7–4510 CPU at 2.60 GHz and 8 GB RAM. In the experi-ments the parameters are set to the values described inTable 4. The parameter settings are obtained by con-ducting several preliminary experiments.

Experiments and evaluationThe goal of the experiments was to evaluate the pro-posed placement architecture using the Best Fit, MaxFit, and ACO-BF scheduling algorithms. The placementarchitecture works as a multitier scheduling algorithm inwhich the placement decision considers the utilizationof both PMs and VMs. Each scheduling algorithm is ex-ecuted 10 times for different number of containers. Ini-tially, all PMs are off with no instantiated VMs. Theoutput of each algorithms is the set of VMs to instanti-ate with their corresponding PMs to turn on.Figure 10 shows the number of instantiated VMs for

different numbers of containers. It is observed that theACO-BF scheduler guarantees the lowest number of in-stantiated VMs compared to the other schedulers. Fur-thermore, the Max Fit algorithm instantiates a largernumber of VMs.

Figures 11 and 12 show the average CPU and memoryutilization of the instantiated VMs using the proposedscheduling algorithms for different numbers of con-tainers. It is clear that the proposed ACO-BF algorithmyields the best resource utilization of VMs.Figure 13 shows the number of PMs turned on for

scheduling different number of containers. It is observedthat the proposed ACO-BF algorithm turns on a largernumber of PMs than the Best Fit and Max Fit algo-rithms. The Max Fit algorithms activates the lowestnumber of PMs.Figures 14 and 15 show the CPU and memory

utilization of the activated PMs for different numbers ofcontainers. The proposed ACO-BF placement algorithmachieves the highest resource utilization. It is clear thatthe Max Fit algorithm guarantees a small number of ac-tive PMs but achieves poor resources utilization, as it ac-tivates the PMs with largest the physical resources. Theproposed ACO-BF placement algorithm activates a largenumber of PMs because it turns on the PMs with thelowest resource capacities that can accommodate the re-sources required by the scheduled containers.Figure 16 shows the time costs of the heuristics, which

are computationally lightweight compared to the

0.5

0.6

0.7

0.8

0.9

1

1.1

25 50 75 100 125 150 175 200 225 250

sM

P detavitca fo noitazilitu yrome

M



Fig. 15 Memory utilization of activated PMs

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

1 2 3 4 5 6 7 8 9 10

Tim

e in

Sec

onds


Max Fit Best Fit

Fig. 16 The time cost of the scheduling heuristics


meta-heuristic ACO-BF, as shown in Fig. 17. Althoughthe ACO requires a substantial amount of time com-pared to the heuristic algorithms, it guarantees the high-est utilization of both VMs and PMs.

Conclusions and future workIn this paper, a placement architecture is presented forcontainer-based tasks on a cloud environment. The pro-posed architecture is a multitier scheduler that aims tomaximize the utilization of both the active PMs and theinstantiated VMs while minimizing the number of activeVMs and PMs. Two scheduling algorithms, includingthe heuristic Max Fit and Best Fit algorithm as well asthe meta-heuristic ACO, are tested using a fitness func-tion that analyzes the percentage of remaining resourcesthat are wasted in the PMs and VMs. The evaluationshows that the ACO is highly effective in terms of VMand PM utilization and in minimizing the number of ac-tive PMs and instantiated VMs. Furthermore, but theproposed ACO increases the number of active PMs be-cause it activates PMs with the lowest configurationsthat can accommodate the required resources.This study assumed that each job consists of inde-

pendent containers. In the future, we will study work-flow containerization processes in which each workflowconsists of a set of dependent communicating container-ized tasks. This type of workflow is represented as a Di-rected Acyclic Graph (DAG), where each containerizedtask requires a specific execution environment to satisfythe priorities and communication constraints of theworkflow. Furthermore, this study considered only thenumber of CPU cores and the memory size as comput-ing resources in the proposed scheduling algorithms.However, in workflow scheduling, additional computingresources that must be considered include processingcores, memory, storage, and data transfers. These add-itional considerations arise because PMs are distributed;thus, data transfer bandwidth is an important factor thatmust be considered in the workflow scheduling problem.Furthermore, maintaining the quality of service (QoS) of

the workflows submitted by the users of the executedcontainers in terms of SLA is another important re-search issue. Finally, additional meta-heuristics such asParticle Swarm and Cuckoo Search, should be tested forscheduling and provisioning workflow containers oncloud environments.

AbbreviationsACO: Ant Colony Optimization; BF: Best Fit; CaaS: Container as a Service;MF: Max Fit; PMs: Physical Machines; VMs: Virtual Machines

AcknowledgementsNot applicable.

FundingNot applicable.

Availability of data and materialsNot applicable.

Authors’ contributionsAll authors read and approved the final manuscript.

Competing interestsThe authors declare that they have no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Author details1Faculty of Computers and Informatics, Suez Canal University, Ismailia, Egypt.2Faculty of Computers and Information Technology, University of Jeddah,Jeddah, Kingdom of Saudi Arabia.

Received: 11 January 2019 Accepted: 13 May 2019

References1. Varghese B, Buyya R (2018) Next generation cloud computing: new trends

and research directions. Future Gener Comput Syst 79:849–8612. Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I (2009) Cloud computing

and emerging IT platforms: vision, hype, and reality for deliveringcomputing as the 5th utility. Future Gener Comput Syst 25:599–616

3. Mell PM, Grance T (2011) SP 800–145. The NIST definition of cloudcomputing. National Institute of Standards & Technology, Gaithersburg, MD

4. Pahl C, Brogi A, Soldani J, Jamshidi P (2018) Cloud container technologies: astate-of-the-art review. IEEE Transactions on Cloud Computing. https://doi.org/10.1109/TCC.2017.2702586

5. What is Docker? Available from: https://www.docker.com/what-docker

0

10

20

30

40

50

60

70

25 50 75 100 125 150 175 200 225 250

Tim

e in

Sec

onds


Fig. 17 The time cost of the proposed metaheuristic ACO


https://doi.org/10.1109/TCC.2017.2702586

https://doi.org/10.1109/TCC.2017.2702586

https://www.docker.com/what-docker

6. Tao Y, Wang X, Xu X, Chen Y (2017) Dynamic resource allocation algorithmfor container-based service computing. In: IEEE 13th internationalsymposium on autonomous decentralized system (ISADS), IEEE, Bangkok,Thailand, 22–24 March 2017

7. Masdari M, Nabavi SS, Ahmadi V (2016) An overview of virtual machineplacement schemes in cloud computing. J Netw Comput Appl 66:106–127

8. Pires FL, Barán B (2015) A virtual machine placement taxonomy. In: 15thIEEE/ACM international symposium on cluster, cloud and grid computing,IEEE, Shenzhen, China, 4-7 May 2015

9. Usmani Z, Singh S (2016) A survey of virtual machine placement techniquesin a cloud data center. Procedia Comput Sci 78:491–498

10. Liu L, Zhe Q (2016) A survey on virtual machine scheduling in cloudcomputing. In: 2nd IEEE international conference on computer andcommunications (ICCC), IEEE, Chengdu, China, 14-17 October 2016

11. Eswaraprasad R, Raja L (2017) A review of virtual machine (VM) resourcescheduling algorithms in cloud computing environment. Journal ofStatistics and Management Systems 20:703–711

12. Dziurzanski P, Indrusiak LS (2018) Value-based allocation of docker containers.In: 26th Euromicro international conference on parallel, distributed andnetwork-based processing (PDP), IEEE, Cambridge, UK, 21-23 March 2018

13. Jaison A, Kavitha N, Janardhanan PS (2016) Docker for optimization ofcassandra NoSQL deployments on node limited clusters. In: Internationalconference on emerging technological trends (ICETT), IEEE, Kollam, India,21–22 October 2016

14. Spicuglia S, Chen LY, Birke R, Binder W (2015) optimizing capacity allocationfor big data applications in cloud datacenters. In: IFIP/IEEE internationalsymposium on integrated network management (IM), IEEE, Ottawa, ON,Canada, 11–15 May 2015

15. Raj VKM, Shriram R (2011) Power aware provisioning in cloud computingenvironment. In: International conference on computer, communicationand electrical technology (ICCCET), IEEE, Tamilnadu, India, 18-19 March 2011

16. Tchana A, Palma ND, Safieddine I, Hagimont D, Diot B, Vuillerme N (2015)Software consolidation as an efficient energy and cost saving solution for aSaaS/PaaS cloud model. In: European conference on parallel processing,Springer, Berlin, Heidelberg, 25 July 2015

17. Yaqub E, Yahyapour R, Wieder P, Jehangiri AI, Lu K, Kotsokalis C (2014)Metaheuristics-based planning and optimization for SLA-aware resourcemanagement in PaaS clouds. In: IEEE/ACM 7th international conference onutility and cloud computing, IEEE, London, UK, 8–11 December 2014

18. Dhyani K, Gualandi S, Cremonesi P (2010) A constraint programming approachfor the service consolidation problem. In: International conference on integrationof artificial intelligence (AI) and operations research (OR) techniques in constraintprogramming, Springer, Berlin, Heidelberg, 14-18 June 2010

19. Dhanashri B, Divya T (2017) Implementation of two level scheduler in cloudcomputing environment. Int J Comput Appl 166:1–4

20. Innocent FM, Alphonsus M, Nansel L, Titus EF, Dashe A (2018) Best-fit virtualmachine placement algorithm for load balancing in a cloud computingenvironment. Int J Sci Eng Res 9:6

21. Zhang R, Zhong A-M, Dong B, Tian F, Li R (2018) Container-VM-PMarchitecture: a novel architecture for docker container placement. In:International conference on cloud computing, Springer InternationalPublishing, Cham, 19 June 2018

22. Shi W, Hong B (2011) Towards profitable virtual machine placement in thedata center. In: Fourth IEEE international conference on utility and cloudcomputing, IEEE, Victoria, NSW, Australia, 5–8 December 2011

23. Dorigo M, Birattari M, Stutzle T (2006) Ant colony optimization. IEEE ComputIntell Mag 1:28–39

24. Xue X, Cheng X, Xu B, Wang H, Jiang C (2010) The basic principle andapplication of ant colony optimization algorithm. In: Internationalconference on artificial intelligence and education (ICAIE), IEEE, Hangzhou,China, 29–30 October 2010

25. Nayyar A, Singh R (2016) Ant colony optimization — computational swarmintelligence technique. In: 3rd international conference on computing forsustainable global development (INDIACom), IEEE, New Delhi, India, 16–18March 2016

26. Zhang J, Hu X, Tan X, Zhong JH, Huang Q (2006) Implementation of an antcolony optimization technique for job shop scheduling problem. Trans InstMeas Control 28:93–108

27. Gao Y, Guan H, Qi Z, Hou Y, Liu L (2013) A multi-objective ant colonysystem algorithm for virtual machine placement in cloud computing. JComput Syst Sci 79:1230–1242

28. Li K, Xu G, Zhao G, Dong Y, Wang D (2011) Cloud task scheduling based onload balancing ant colony optimization. In: Sixth annual chinagridconference, IEEE, Liaoning, China, 22–23 August 2011

29. Mishra R, Jaiswal A (2012) Ant colony optimization: a solution of load balancingin cloud. International Journal of Web & Semantic Technology 3:33–50

30. Nishant K, Sharma P, Krishna V, Gupta C, Singh KP, Nitin, Rastogi R (2012)Load balancing of nodes in cloud using ant colony optimization. In: UKSim14th international conference on computer modelling and simulation, IEEE,Cambridge, UK, 28–30 March 2012

31. Reiss C, Wilkes J, Hellerstein JL (2011) Google cluster-usage traces: format+schema. Technical Report, Google Inc., Mountain View, CA, USA


a placement architecture for a container as a service ... · research open access a placement...

Documents