SOCA (2017) 11:121–135DOI 10.1007/s11761-016-0201-x

ORIGINAL RESEARCH PAPER

Performance-aware server architecture recommendationand automatic performance verification technology on IaaS cloud

Yoji Yamato1

Received: 25 April 2016 / Revised: 4 September 2016 / Accepted: 23 October 2016 / Published online: 2 November 2016© The Author(s) 2016. This article is published with open access at Springerlink.com

Abstract In this paper, we propose a server architecture rec-ommendation and automatic performance verification tech-nology, which recommends and verifies appropriate serverarchitecture on Infrastructure as a Service (IaaS) cloud withbaremetal servers, container-based virtual servers and virtualmachines. Recently, cloud services are spread, and providersprovide not only virtual machines but also bare metal serversand container-based virtual servers. However, users need todesign appropriate server architecture for their requirementsbased on three types of server performances, and users needmuch technical knowledge to optimize their system per-formance. Therefore, we study a technology that satisfiesusers’ performance requirements on these three types of IaaScloud. Firstly, we measure performance and start-up timeof a bare metal server, Docker containers, KVM (Kernel-based Virtual Machine) virtual machines on OpenStack withchanging number of virtual servers. Secondly, we proposea server architecture recommendation technology based onthe measured quantitative data. A server architecture rec-ommendation technology receives an abstract template ofOpenStackHeat and function/performance requirements andthen creates a concrete template with server specificationinformation. Thirdly, we propose an automatic performanceverification technology that executes necessary performancetests automatically on provisioned user environments accord-ing to the template. We implement proposed technologies,confirm performance and show the effectiveness.

B Yoji Yamatoyamato.yoji@lab.ntt.co.jp

1 Software Innovation Center, NTT Corporation, 3-9-11Midori-cho, Musashino-shi 180-8585, Japan

Keywords Performance · Cloud computing · IaaS · Baremetal · Container · Hypervisor · OpenStack · Heat ·Automatic test

1 Introduction

Infrastructure as a Service (IaaS) cloud services haveadvanced recently, and users can use virtual resources such asvirtual servers, virtual networks and virtual routes on demandfrom IaaS service providers (for example, Rackspace pub-lic cloud [1]). Users can install OS and middleware such asDBMS, Web servers, application servers and mail serversto virtual servers by themselves. And open-source IaaS soft-ware also becomesmajor, and adoptions ofOpenStack [2] areincreasing especially. Our companyNTT group has launchedproduction IaaS services based on OpenStack since 2013[3].

Most cloud services provide virtual computer resourcesfor users by virtual machines on hypervisors such as Xen[4] and Kernel-based Virtual Machine (KVM) [5]. However,hypervisors have demerits of much virtualization over-head. Therefore, some providers start to provide container-based virtual servers (hereinafter, containers), which per-formance degradations are little, and bare metal servers(hereinafter, baremetal), which does not virtualize a physicalserver.

Providing alternatives of bare metals, containers and vir-tual machines to users can enhance IaaS adoptions, we think.It is generally said that bare metals and containers show bet-ter performance than virtual machines, but an appropriateusage is not mature based on three types of server perfor-mances. Therefore, when providers only provide these threetypes of servers, users need to design appropriate serverarchitecture for their performance requirements and need

123

122 SOCA (2017) 11:121–135

Fig. 1 OpenStack architectureprogram

browser

Cinder Swift

Nova

REST APIHorizon

Neutron

Glance

storage

Server

Virtual Network

OpenStack

Ironic HeatKey-stone

Ceilo-meter

much technical knowledge to optimize their system perfor-mance.

Therefore, to reduce users’ efforts of designing andverifying, we propose a technology that satisfies users’ per-formance requirements on these three types of IaaS cloud.Firstly, we measure performance and start-up time of a baremetal server provisioned by Ironic [6] and Docker [7] con-tainers, KVM virtual machines on OpenStack with changingnumber of virtual servers. Secondly, we propose a serverarchitecture recommendation technology based on the mea-sured quantitative data. In OpenStack, Heat [8] provisionsvirtual environments based on text format templates.A serverarchitecture recommendation technology receives an abstracttemplate ofHeat and function/performance requirements andthen creates a concrete template with server specificationinformation. Thirdly, we propose an automatic performanceverification technology, which executes necessary perfor-mance tests automatically on provisioned user environmentsbased on the template to guarantee performance. We imple-ment proposed technologies, confirm performance and showthe effectiveness.

The rest of this paper is organized as follows:We introducean IaaS platform OpenStack, review three types of serversand clarify problems in Sect. 2. We measure performanceof the three types of servers on OpenStack and discuss anappropriate usage in Sect. 3. We propose a server archi-tecture recommendation technology, which satisfies users’requirements, and an automatic performance verificationtechnology, which confirms performance on the provisionedenvironments in Sect. 4. We confirm the performance ofproposed technologies in Sect. 5. We compare our work toother related works in Sect. 6. We summarize the paper inSect. 7.

2 Overview of existing technologies

2.1 Outline of OpenStack

OpenStack [2], CloudStack [9] and Amazon Web Services(AWS) [10] are major IaaS platforms. The basic idea of ourproposed technologies is independent from IaaS platforms.For the first step, however, we implement a prototype ofthe proposed technologies on OpenStack. Therefore, we useOpenStack as an example of an IaaS platform in this sub-section. Because OpenStack community would like to catchup AWS currently, OpenStack major functions are similar tothose of AWS.

OpenStack is composed of function blocks that man-age each virtual resource and function blocks that integrateother function blocks. Figure 1 shows a diagram of Open-Stack function blocks. Neutron manages virtual networks.OVS (Open Virtual Switch) [11] and other software switchescan be used as virtual switches. Nova manages virtualservers. Main servers are virtual machines on hypervisorssuch as KVM. But Nova also can control containers suchas Docker containers and bare metal servers provisioned byIronic. OpenStack provides two storage management func-tion blocks: Cinder for block storage and Swift for objectstorage. Glance manages image files for virtual servers. Heatorchestrates these function blocks and provisions multiplevirtual resources according to a template text file. A templateis a description of server and virtual resource configurationfor orchestration. Ceilometer is a monitoring function of vir-tual resource usage. Keystone is a function block that enablessingle sign-on authentication among other OpenStack func-tion blocks. The functions of OpenStack are used throughREST (Representational State Transfer) APIs. There is also

123

SOCA (2017) 11:121–135 123

a Web GUI called Horizon that uses the functions of Open-Stack.

2.2 Qualitative comparison of bare metal, container,hypervisor

In this subsection, we compare bare metal, container andhypervisor qualitatively.

Bare metal is a non-virtualized physical server and sameas an existing dedicated hosting server. IBM SoftLayer pro-vides bare metal cloud services adding characteristics ofprompt provisioning and pay-per-use billing to dedicatedservers. InOpenStack, Ironic component provides baremetalprovisioning. Because bare metal is a dedicated server,flexibility and performance are high, but provisioning andstart-up time are long, and it also cannot conduct live migra-tions.

Containers’ technology is OS virtualization. OpenVZ[12] or FreeBSD jail was used for VPS (Virtual PrivateServer) [13] for many years. Computer resources are iso-lated with each unit called container, but OS kernel is sharedamong all containers. Docker that uses LXC (Linux Con-tainer) appeared in 2013 and attracted many users becauseof its usability. Containers do not have kernel flexibility,but a container creation only needs a process invoca-tion, and it takes a short time for start-up. Virtualizationoverhead is also small. OpenVZ can conduct live migra-tions, but Docker or LXC cannot conduct live migrationsnow.

Hypervisors’ technology is hardware virtualization, andvirtual machines are behaved on emulated hardware; thus,users can customize virtual machine OS flexibly. Majorhypervisors are Xen, KVM and VMware ESX. Virtualmachines have merits of flexible OS and live migrations, butthose have demerits of performance and start-up time.

However, because these characteristics are only qual-itative ones, we evaluate performance and start-up timequantitatively in Sect. 3.

2.3 Problems of multiple types of IaaS serverprovisioning

Here, we clarify a problem of three types of IaaS server pro-visioning.

Three types of servers increase options of price and per-formance for users. It is generally said that bare metals andcontainers show better performance than virtual machines onhypervisors. However, there are few works to compare per-formance and start-up time of those three in same conditions,and appropriate usage discussions based on quantitative dataare not mature. For example, Fester et al. [14] compared theperformance of bare metal, Docker and KVM, but there areno data of start-up time or performance with changing num-

ber of virtual servers. Therefore,whenproviders only providethese three types of servers, users need to select and designappropriate server architecture for their performance require-ments and need much technical knowledge or performanceevaluation efforts to optimize their system performance. Thisis not only on OpenStack, but also on AWS and other IaaSplatforms for users to design appropriate server architecturefor their requirements.

There are some works of resource allocation on cloud ser-vices to use server resources effectively (for example, [15]);these technologies’ targets are mainly to reduce providerscost such as energy usage by allocating virtual machinesappropriately. On the other hand, a technology that selectsappropriate type servers from multiple types based on users’performance and other requirements is not sufficient. There-fore, we study an appropriate type server recommendationtechnology in Sect. 4 using performance data of Sect. 3.

Note that a smooth migration among these three types ofservers is another problem. Live migrations cannot be donebetween different platforms, migrations need steps of imageextraction and image deployment. For example, VMwareprovides a migration tool that helps a migration from otherhypervisors toVMwareESX, and it extracts images, convertsimages then deploys images [16]. In this paper, migrationsare out of scope because we use existing vendors’ migrationtools.

3 Performance comparison of bare metal, Dockerand KVM

This sectionmeasures performance and start-up time of threetypes of servers with same conditions. We use OpenStackversion Juno as a cloud controller, a physical server provi-sioned by Ironic as bare metal, Docker 1.4.1 as a containertechnology and KVM/QEMU 2.0.0 as a hypervisor. Ironic,Docker and KVM are de facto standard software in Open-Stack community. Server instances are Ubuntu 14.04 LinuxserverswithApache2Web servers from10GB imagefile, andwe request three types of instances provisioning to a samephysical server using OpenStack compute component Nova.

Functional characteristic comparisons are generally dis-cussed whether we can conduct a live migration or not,whether we can customize a kernel or not, whether wecan scale server resources with short time or not. However,regarding quantitative comparison of performance and start-up time, there is no work when we change number of servers.The work of Fester et al. [14] only compares performance ofthree types of serverswith fixed one server. The papers of Seoet al. [17] andMorabito et al. [18] also compare performancesof two or three types of servers by several benchmarks. Inthis section, we confirm server performance and start-up timewhen number of servers is changed.

123

124 SOCA (2017) 11:121–135

Fig. 2 Performancemeasurement servers’specifications

Management server Server for performance measurement

L2 SwitchOne is IPMI port for

Ironic PXE boot

3.1 Performance measurement items

– Measured servers: bare metal provisioned by Ironic, con-tainers based on Docker, Virtual machines deployed onKVM.

– Number of virtual servers: 1, 2, 3, 4

Only 1 for bare metal case, 1–4 containers for Dockercase and 1–4 virtual machines for KVM case. When therearemultiple virtual servers, all physical resources are equallyseparated to these multiple servers.

– Performance measurement.

UnixBench [19] is conducted to acquire UnixBench per-formance indexes. Note that UnixBench is a major sys-tem performance benchmark. When we measure multipleserver performance, UnixBench is conducted concurrently.It should be noted that UnixBench measures more than 10items such as Double-Precision Whetstone, File Copy andProcess Creation to score Index Values.

– Start-up time measurement.

A time from Nova server instance creation API call to eachLinux and Apache2 server start-up is measured. When wemeasure multiple server start-up time, instance creation APIis called concurrently. For bare metal case, we measure notonly total time but also each processing time of start-up, andwe also measure the 1st time boot and the 2nd time boot.

3.2 Performance measurement environment

For a performance measurement environment, we preparedone physical server on which three types of servers were

provisioned and one physical server which had OpenStackcomponents (Nova, Ironic, PXE server for Ironic PXE bootand so on). These servers were connected with GigabitEthernet and Layer 2 switch. Figure 2 shows each serverspecification.

3.3 Performance of bare metal, Docker and KVMmeasurement environment

3.3.1 UnixBench performance

Figure 3 shows a performance comparison of three types ofservers. Vertical axis shows UnixBench performance indexvalue, and horizon axis shows each server with changingnumber of virtual servers. Showed index values are averageof 3 time measurements. We omit performance result of eachitem of UnixBench such as File Copy.

Based on Fig. 3 results, it is clear that Docker containersperformance degradation is about 75% performance com-pared to bare metal performance. And it is also said thatDocker performance is degraded when we change numberof virtual servers, but it is not inverse proportion. When wesee each item, almost all performances of measured items ofDocker are better than KVMbut File Copy performance is assame as KVM. Therefore, the total index value of Docker isabout 30% better than that of KVMwhen number of server is1. KVM virtual machines performance degradation is largerthan Docker containers and only 60% performance com-pared to bare metal performance when number of server is 1.However, KVM virtual machines performance degradationtendency with changing number of virtual servers is as sameas Docker containers. And it is also said that the improve-ment in Docker performance compared to KVM becomeslittle when number of servers is increased.

123

SOCA (2017) 11:121–135 125

Fig. 3 UnixBench performanceindex score comparison

0

500

1000

1500

2000

2500

3000

3500

4000

1server

1server

2servers

3servers

4servers

1server

2servers

3servers

4servers

Baremetal Docker KVM

(a) (b)

(sec

)

(sec

)

1st time 2nd time0

100

200

300

400

500

600 Service star�ng

PXE boot

Image wri�ng

PXE deploy

API processing

0

30

60

90

120

150

180

455

520

1st �

me

2nd

�me

1 se

rver

2 se

rver

s

3 se

rver

s

4 se

rver

s

1 se

rver

2 se

rver

s

3 se

rver

s

4 se

rver

s

Baremetal Docker KVM

Fig. 4 Start-up time comparison. a Bare metal, Docker and KVM start-up time. b Each processing time of bare metal start-up

Virtual machine and container performance depend oneach virtualization technology of hypervisor and container.Therefore, basically UnixBench performance is not differedin other IaaS platforms such as CloudStack.

3.3.2 Start-up time

Figure 4a shows start-up time of three types of servers.Showed start-up times are average of 3 time measurements.When virtual servers are multiple, average start-up time isshowed. Figure 4b shows each processing time of bare metalstart-up for the 1st time boot and the 2nd time boot. FromFig. 4a, bare metal start-up takes much long time than KVMand Docker. This is because bare metal start-up needs imagewriting for PXE boot for the 1st time boot and it takes longtime. For the 2nd time boot, it does not need image writingand total start-up time is about only 200s (see, Fig. 4b).

Comparing Docker and KVM, Docker containers start-upis shorter than KVM virtual machines and is less than 15s.This is because a virtual machine start-up needs OS boot, buta container creation only needs a process invocation. Pre-

cisely, Docker instance creation only takes several hundredms, but OpenStack processing such as API check, port cre-ation and IP address setting take about 5 s.

Implementation of virtual resource creation API is differ-ent in each IaaS platform; start-up time may be differed littleon other IaaS platforms, but most of start-up time dependson eachvirtualization technologyof hypervisor and container(virtual machine creation or container process invocation).

3.4 Discussion

Here, we discuss appropriate usages of IaaS servers basedon quantitative data. Because bare metal shows better per-formance than other two types servers, it is suitable to uselarge-scale DB processing or real-time processing, whichhave performance problems when we use virtual machines.Containers lack flexibility of kernel, but performance degra-dation is small and start-up time is short. Thus, it is suitablefor auto-scaling for existing servers or shared usages of basicservices such as Web or mail. Hypervisors are suitable to

123

126 SOCA (2017) 11:121–135

Fig. 5 Processing steps ofproposed method

use for areas, which need system flexibility such as businessapplications on specific OS.

Of course, UnixBench and start-up times are not onlyways to compare performances. Various performances maybe needed in production service phase of our proposed sys-tem. Regarding benchmarks, CPU/memory performancescan be evaluated by UnixBench or GeekBench, networkperformance can be evaluated by nuttcp or NDT, storage per-formance can be evaluated by hdparm or fio, DB transactionperformance can be evaluated by TPC (Transaction Process-ing Performance Council) and energy consumption can beevaluated by LINPACK benchmark of Flops/Watt. These arecandidates to measure by cloud providers.

4 Proposal of automatic verification technology ofvirtual machines patches

We propose a technology that enables a provider recom-mends appropriate server architecture and verifies it based onusers performance requirement in this section. In Sect. 4.1,we explain the steps of server architecture recommendationand automatic performance verification. The figure showsOpenStack, but OpenStack is not a precondition of the pro-posed method. In Sect. 4.2, we explain the process of serverarchitecture recommendationusingSect. 3 performancedata,which is one of core process of these steps. In Sect. 4.3, weexplain the process of performance test extraction for provi-sioned user environment to guarantee performance, which isanother core process of these steps.

4.1 Processing steps

Our proposed system is composed of Server architecture rec-ommendation and Automatic verification Functions (here-inafter SAFs), a test case DB, Jenkins and an IaaS controllersuch as OpenStack. Figure 5 shows the processing steps ofserver architecture recommendation and automatic verifica-tion. There are eight steps in our proposal.

1. A user specifies an abstract template and requirementsto SAFs. A template is a JSON text file with virtual resourceconfiguration information and is used by OpenStack Heat[8] or Amazon CloudFormation [20] to provision virtualresources in one batch process. Although Heat templateneeds server flavor (=specification) information, an abstracttemplate does not include flavor information. A template alsodescribes image files for server deployments. Here, SAFsonly recommend IaaS layer server architecture, and PaaSlayer recommendation is currently out of scope. If userswould like to deploy PaaS such as Deis, Flynn or CloudFoundry, users need to specify them as image files.

A user also specifies requirements that include eachserver functional requirements and non-functional require-ments of each server performance and total price. Functionalrequirements are that OS are normal Linux or non-Linux orcustomized Linux and are used to judge whether a containersatisfies requirements. Performance requirements are lowerlimit of server performance such as throughput.

Note that if a user would like to replicate existing virtualenvironment, we can use a technology of Yamato et al. [21]to extract a template of existing environment.

123

SOCA (2017) 11:121–135 127

Fig. 6 Example of Web 3-tierconnection pattern

2. SAFsunderstand server connectionpattern and installedsoftware from a template and image files specified by a user.If there is a user original image file, SAFs need to get infor-mation from a volume, which is deployed by the image tounderstand what software is installed. In this case, a userneeds to input login information in step 1. After analyzing atemplate and images, SAFs recognize a system configuration(for example, Fig. 6 configuration).

3. SAFs select server types and recommend server archi-tecture using user requirements specified in step 1. Becausethis is a first core step of proposed method, we explain itin detail in Sect. 4.2. When SAFs recommend server archi-tecture, SAFs add a specific flavor for each server to Heattemplate. Thus, a user can distinct each server type as baremetal or container or virtual machine by flavor descriptions.

4. A user confirms the recommendation and replies anacknowledgment to SAFs. If the user does not satisfythe recommendation, the user may reject the recommen-dation and modify a concrete template. SAFs propose aconcrete template based on specified function and perfor-mance requirements, but decision factors may include othercriteria such as ease of use, experiences and interoperabil-ity.

After acknowledgment or user’s modification of concretetemplate, SAFs fix a concrete template with each server fla-vor.

5. SAFs request an IaaS controller to deploy the concretetemplate with the target tenant. An IaaS controller provisionsvirtual resources of the user environment on the specifiedtenant.

6. SAFs select appropriate performance verification testcases from the test case DB to show a sufficient performanceof user environment provisioned based on the template. SAFsselect test cases not only each individual server performancebut also multiple servers’ performance such as transaction

processing of Web 3-tier model (for example, Fig. 6 is oneof Web 3-tier model). Because this is a second core step ofproposed method, we explain it in detail in Sect. 4.3

7. SAFs execute performance test cases selected in Step6. We use an existing tool, Jenkins [22], to execute test casesselected from the test case DB. Although performance ver-ification is targeted for servers, verification test cases areexecuted for all virtual resources in a user environment. In acase where virtual machines with Web servers are under onevirtual load balancer, Web server performance needs to betested via the virtual load balancer.

8. SAFs collect the results of test cases for each user envi-ronment using Jenkins functions. Collected data are sent tousers via mail or Web. Users evaluate system performanceby these data and judge to start using IaaS cloud. Users needto understand the performance result which may be degradedwhen other users’ VMs or containers in same nodes usemuchcomputer resources. If users do not satisfy the performanceresults, users can release the environments.

4.2 Server architecture recommendation technology

In this subsection, we explain in detail step 3 of server archi-tecture recommendation, which is a first core step of ourproposal. SAFs understand server connection pattern andinstalled software from a template and image files speci-fied by a user. Users’ non-functional requirements specifylower limit of performance such as UnixBench Index valueor each measured item value [(e.g., Double-Precision Whet-stone > 1500MWIPS (Million Whetstones Instructions PerSecond)] and higher limit of total system price (e.g., Totalprice < 2000USD/month). Based on user requests, SAFsselect servers from a resource pool of bare metal, virtualmachine and container servers. It should be noted that func-tional and performance requirements are for each server

123

128 SOCA (2017) 11:121–135

selection, but total system price requirement is for checkingall selected servers.

Generally, server prices are container< virtualmachine<bare metal. Therefore, the basic selection logic is that SAFsselect lower price servers when those servers satisfy users’requirements.

Firstly, SAFs select bare metal servers, which need highperformance. Throughput or other thresholds are determinedby Sect. 3 performance results. If user performance require-ments specified in step 1 exceed thresholds such as 3000 ofUnixBench Index score, SAFs select bare metal servers. Forexample, because order management DB of Web shoppingsystem needs strong consistency and is difficult for parallelprocessing, bare metal is appropriate when a system is abovea certain scale. If a system does not require strong consis-tency and allows Eventual Consistency [23], a container orvirtualmachine becomes alternatives for aDB server becausedistributed Key-Values store such as memcached [24] can beadopted to enhance throughput.

Next, SAFs narrow down server type by OS requirements.SAFs check function requirements whether a server OS isnormal OS or customized OS and select a virtual machinefor latter case. In 2015, Microsoft announced Hyper-V Con-tainer; thus, we can use containers not only for Linux OS butalso for Windows OS.

Lastly, SAFs select containers for servers which OS arenormal OS.

Figure 7 shows a server selection logic flow of proposedmethod. After each server selection, SAFs select remainedservers described in an abstract template recursively. Afterall servers are selected, SAFs check total price of selectedcandidate servers. If total price limit is satisfied, SAFs replyusers with a concrete template of selected servers. If totalprice limit is not satisfied, SAFs may change one server toother lower price server or may ask users to change.

When physical servers are heterogeneous, performancesare differed according to not only provision type but alsophysical server type. Therefore, cloud providers need tomeasure performances for each server type beforehand. Forexamples of heterogeneous servers, there are servers withmany core CPU and servers with strong GPU. And if a newvirtualization technology is appeared, performance evalua-tion is also needed. When there are heterogeneous physicalservers and new virtualization technologies, SAFs selectservers similar to Fig. 7. Based on user requests with func-tional, performance and price requirements, SAFs selecta candidate server which satisfies functional/performancerequirements with lower price. But, note that Fig. 7 flow addsmore branches for heterogeneous servers or new virtualiza-tion technology to select themwhen those servers appropriateto user requests in individual server selection phase.

Figure 7 flow is for a system construction phase. Thereare APIs to manage Heat stack by templates, stack-create

and stack-update. In construction phase, we use stack-create API. However, for stack-update during operationphase such as enhancing scalability for increasing usages,the start-up time branch is added to the flowchart. Ifan added server requirement of start-up time to enableApache2 is less than 15s, we need to select contain-ers. To enhance system scalability (e.g., for temporal Webaccess during the event), containers are good options to beselected.

4.3 Automatic performance verification technology

In this subsection, we explain in detail step 6 of performancetest case extraction, which is a second core step of our pro-posal.

Previously, we had developed an automatic patch ver-ification function for periodical virtual machine patches[25]. A key idea of test case extractions of Yamato [25]is 2-tier software abstracting to reduce prepared test cases.The work of Yamato [25] stores relations of softwareand software group, which is a concept grouping differentversions of software, and function group, which is con-cept grouping same functions software, and it extracts testcases corresponding to upper tier concept. For example, incase of MySQL 5.6 installed on virtual machines, Yam-ato [25] method executes DB function group test cases andMySQL software group test cases. This idea has a meritfor operators not to prepare each software regression testcases.

However, Yamato [25] can extract only unit regressiontests because it selects test cases corresponding to each vir-tual server software. The problem is that it cannot extractperformance tests with multiple virtual servers.

To enable performance tests with multiple servers, wepropose a performance test extraction method for each con-nection pattern of servers using information of Heat templateconnection relation and installed software.

Firstly, proposed method stores software information intest case DB not only Yamato’s [25] software relation infor-mation of Table 1a but also connection pattern informationof Table 1b. Here, Table 1b second row shows that “con-nection pattern” is Web 3-tier and “deployment config” is{Web, AP}{DB}. A deployment config of {Web, AP}{DB}means one server has aWeb server and an Application serverand another server has a DB server. For example, connectionrelations like Fig. 6 can be analyzed by parsing a Heat JSONtemplate description in step 2. Using connection relations oftemplates, installed software and Table 1a software relationdata, user server deployment configurations can be judged as{Web, AP}{DB}. Adding Table 1b connection pattern infor-mation, a connection pattern also can be judged asWeb 3-tiermodel.

123

SOCA (2017) 11:121–135 129

Need for high performance of

bare metal?

Select bare metal server

Need for customize OS?

Select virtual machine Select container server

No

Yes

Yes

No

Reply concrete template

Receive abstract template

Satisfy price limit of user request?

Are there remained servers

to select?

Yes

No

YesNo

Reply user or change one server

Individual server selection phase

Total systemcheck phase

Fig. 7 Server type selection flow for initial server construction

Next, proposed method adds a “connection pattern” col-umn to [25]’s test case information of Table 2 and enablesto define test cases corresponding to each connection pat-tern. For example, Table 2 fourth row shows that TPC-Cbenchmark [26] test can be used for regression tests for Web3-tier connection pattern. There are following performancetest case examples in addition to TPC-C. Performances ofmail systems that may consist of several servers can be eval-uated by SPEC MAIL2001. Big data analysis performancesof Hadoop cluster can be evaluated by TestDFSIO and Test-Sort. Web performances of Apache server can be evaluatedby Apache Bench.

By these improvements, SAFs judge connection patternsby templates created in step 3 and installed software extractedin step 2. For example of Fig. 6, Tables1, 2 case, SAFs judgea connection pattern as Web 3-tier. Then, when SAFs extracttest cases in step 6, those extract not only each server Webor DB performance test cases but also TPC-C test for Web3-tier connection pattern.

It should be noted that our proposed technology conductsperformance test cases from prepared common test casesfor installed software and server connection patterns. There-

fore, SAFs do not conduct performance tests fully matchedapplications which users run. These application performancetests are conducted by users, after users confirm commonperformance test results and judge of servers utilizationstart. But, to increase satisfactions of users’ performancetest demands, cloud providers may increase common testcases of test case DB or may conduct users’ application per-formance test cases which are provided by users instead ofusers.

5 Performance evaluation of proposed method

We implemented server architecture recommendation andautomatic performance verification functions of Fig. 6. Weimplemented the system on OpenStack Folsom. It is imple-mented on Ubuntu 12.04 OS, Tomcat 6.0, Jenkins 1.5 byPython 2.7.3.

We confirmed correct behaviors of proposed functions byselecting and executingWeb 3-tier performance test of TPC-C for {Web}{AP}{DB}, {Web,AP}{DB}or {Web,AP,DB}deployment configurations of Web 3-tier virtual servers.

123

130 SOCA (2017) 11:121–135

Table 1 (a) Software relationdata, (b) connection pattern data

Function group Software group Software

(a)

OS Windows Windows Server 2012

OS Windows Windows 8.1

OS RHEL RHEL 7.0

OS RHEL RHEL 6.1

DB Oracle Oracle11g

DB Oracle Oracle 10g

DB MySQL MySQL 5.0

DB MySQL MySQL 4.0

Web Apache Apache 2.1

Web Apache Apache 2.2

AP Tomcat Tomcat 6.0

AP Tomcat Tomcat 7.0

Connection pattern Deployment config

(b)

Web 3-tier {Web}{AP}{DB}

Web 3-tier {Web, AP}{DB}

Web 3-tier {Web}{AP, DB}

Web 3-tier {Web, AP, DB}

Table 2 Test case data

Connection pattern Function group Software group Software Test case Test case class

DB Table CRUD DB function group

DB character garbling check DB function group

DB MySQL Access by phpMyAdmin MySQL software group

Web Apache Apache Bench Apache software group

Web 3-tier TPC-C benchmark test Web 3-tier connection pattern

Mail system SPEC MAIL2001 Mail connection pattern

Hadoop cluster TestDFSIO Hadoop cluster connection pat-tern

Hadoop cluster TestSort Hadoop cluster connection pat-tern

Our technology creates virtual resources based on a Heatconcrete template and executes performance verification testcases. We evaluated the performance of the total process-ing time and each section processing time with changing thenumber of concurrent processing threads.

5.1 Performance measurement conditions

Processing steps to bemeasured: template deployment, testerresource preparation such as Internet connection settings, testcase selection and test case execution.

User tenant configuration:

– Each user tenant has two virtual machines, two volumes,two virtual Layer-2 networks, and one virtual router.

– Each virtual machine’s specifications are one CPU withone Core, 1 GB RAM, and one attached volume with asize of 10 GB, and the installed OS is CentOS 6.

– Apache 2.1 and Tomcat 6.0 are installed on one volume,and MySQL 5.6 is installed on one volume for virtualmachines’ software.

Selected test case: TPC-C test case.

123

SOCA (2017) 11:121–135 131

legend

DMZ-Load Balancer KVMGlance application server

KVMGlance application server

Internal-Load Balancer

KVM

OpenStack -Volume

OpenStack -Volume

OpenStack -Hypervisor

OpenStack -Hypervisor

DB (OpenStack&Test Case)

OpenStack -Network

OpenStack -Network

Physical Server

Virtual Server

Internet segment Control segment

KVM

User terminal

Operator terminal

Template serverfor tenant replication

Openstack API server

KVM

Template serverfor tenant replication

Openstack API server

Structure proposal &automatic verificationserver

Fig. 8 Test environment for proposed method confirmation

Number of concurrent processing threads: 1, 3, 5Automatic verifications are started by a command line

interface, and parallel concurrent processing is managed bya script in this performance measurement.

5.2 Performance measurement environment

Figure 8 shows the test environment. Meanwhile, there aremany servers for OpenStack virtual resources; the mainserver of thismeasurement is an automatic verification server.These servers are connected with Gigabit Ethernet.

In detail, Fig. 8 shows the physical and virtual serversand the modules in each server. For example, in the Open-Stack API server case, this server is a virtual server, it is inboth the Internet segment and the Control segment, and itsmodules are a Cinder scheduler, Cinder API, nova-api, key-stone, glance-registry and nova-scheduler. Two servers areused for redundancy. Other servers are the proposed auto-matic verification server, a user terminal and an operatorterminal, Glance application servers for image upload, NFSstorage for images, template servers for tenant replication,a DB for OpenStack and test cases, OpenStack servers for

virtual resources, iSCSI storage for the data of these servers,and load balancers for load balancing.

Table 3 lists the specifications and usage for each server.For example, in the DB case (6th row), the hardware is HPProLiant BL460cG1, the server is a physical server, the nameis DB, the main usage is OpenStack and Test case DB, theCPU is a Quad-Core Intel Xeon 1600MHz×2 and the num-ber of cores is 8, RAM is 24 GB, the assigned HDD is 72GB, and there are four NICs (Network Interface Cards).

5.3 Performance measurement results

Figure 9 shows each processing time of automatic per-formance verification. When there were several threads ofconcurrent processing, the average processing time is shown.In all cases of concurrent processing (1, 3 and 5), the tem-plate deployment takes a lot of time, while performance testcase execution takes only 15min. Most of these times arecopy times fromGlance to Cinder of OpenStack side. Even ifOpenStackvirtual resource provisioning takesmuch time,wecan complete within one and half hour with 3 of concurrentprocessing threads from batch provisioning to performanceevaluations. If users design server architecture and measure

123

132 SOCA (2017) 11:121–135

Table3

Server

specificatio

nsof

testenvironm

ent

Hardw

are

Physicalor

VM

Nam

eMainusage

CPU

RAM

(GB)

HDD

NIC

Mod

elname

Core

logical(GB)

HPProLiant

BL46

0cG6

Physical

KVM

host

Qua

d-CoreIntel

Xeon25

33MHz×

28

4830

04

VM

OpenS

tack

API

server

OpenS

tack

stateless

processsuch

asAPL

server

Assign:

4Assign:

8Assign:

60

VM

Templateserver

Template

managem

entfor

tenant

replication

Assign:

4Assign:

8Assign:

60

HPProLiant

BL46

0cG6

Physical

KVM

host

Qua

d-CoreIntel

Xeon25

33MHz×

28

4830

04

VM

Glanceapplication

server

Receive

requests

relatedto

glance

Assign:

8Assign:

32Assign:

150

HPProLiant

BL46

0cG1

Physical

DB

OpenS

tack

andtest

case

DB

Qua

d-CoreIntel

Xeon16

00MHz×

28

2472

4

HPProLiant

BI460

cG1

Physical

OpenS

tack-

Network

used

forOpenS

tack

logicaln

etwork

resources

Qua

d-CoreIntel

Xeon16

00MHz×

28

1872

6

HPProLiant

BI460

cG1

Physical

OpenS

tack-

Volum

eused

forOpenS

tack

logicalv

olum

eresources

Qua

d-CoreIntel

Xeon16

00MHz×

28

1872

6

HPProLiant

BL46

0cG1

Physical

OpenS

tack-

Hyp

ervisor

used

forOpenS

tack

VM

resources

Qua

d-CoreIntel

Xeon16

00MHz×

28

2472

4

IBM

HS2

1Ph

ysical

Structure

prop

osal

and

automatic

verification

server

Proposed

architecture

recommendatio

nandautomatic

verification

XeonE51

603.0G

Hz×

12

272

1

IBM

HS2

1Ph

ysical

DMZ-L

oad

balancer

Loadbalancer

for

internetaccess

XeonE51

603.0G

Hz×

12

272

1

IBM

HS2

1Ph

ysical

Internal-loa

dba

lancer

Loadbalancer

for

internalaccess

XeonE51

603.0G

Hz×

12

272

1

IBM

HS2

1Ph

ysical

KVM

host

XeonE51

603.0G

Hz×

12

272

1

VM

UserVM

VM

foruser

term

inal

Assign:

1Assign:

1Assign:

20

VM

OperatorVM

VM

forop

erator

term

inal

Assign:

1Assign:

1Assign:

20

EMCVNX53

00Ph

ysical

iSCSI

storage

iSCSI

storageforuser

volume

500

EMCVNX53

00Ph

ysical

NFSstorag

eNFS

storageforimage

500

123

SOCA (2017) 11:121–135 133

Fig. 9 Processing times of proposed automatic verification

performance based on their performance requirements bythemselves, it takes much more time. Through this result, weevaluate our proposed technologies are effective to reduceusers’ efforts.

6 Related work

Like OpenStack, OpenNebula [27] and CloudStack [9]are open-source Cloud software. OpenNebula is a virtualinfrastructure manager of IaaS building. OpenNebula man-ages VM, storage, network of company and virtualizessystem resources to provide Cloud services. Our group alsocontributes to developments of OpenStack itself. Some bugfixes and enhancements of OpenStack are our group contri-butions [28,29].

When we use other IaaS platforms like CloudStack andAmazon Web Services, some efforts are needed. Regard-ing performance comparisons, start-up times depend on eachIaaS platform because start-up is executed via each IaaS plat-form’s API. Therefore, we need to measure start-up times oneach IaaS platforms. Regarding batch provisioning, CloudFormation [18] is a template base provisioning technologyon Amazon Web Services, and CloudStack also has a tem-plate base provisioning technology. Therefore, SAFs can usebatch provisioning by implementing functions to parse eachdifferent format template and call each IaaS platform API oftemplate deployment.

The paper [15] is a research of dynamic resource allo-cation on cloud. This work allocates data center resourcesbased on application demands and support green computingby optimizing the number of servers in use. There are someworks of resource allocation on cloud services to use serverresources effectively keeping user policy [30,31]. As same as

Xiao et al. [15], ourwork is also a resource allocation technol-ogy on OpenStack, but our work targets to resolve problemsof appropriate server type selection from 3 types of servers.There is no similar technology to recommend appropriateserver architecture on IaaS cloud with bare metals, contain-ers and virtual machines.

The work of Fester et al. [14] compared performance ofbare metal, Docker and KVM. However, there are no dataof start-up time or performance with changing number ofvirtual servers, and appropriate usage discussions of threetypes of servers are not mature. We measured performanceof a bare metal provisioned by Ironic, Docker containers andKVM virtual machines with same conditions and evaluatedquantitatively.

AmazonCloudFormation [20] andOpenStackHeat [8] aremajor template deployment technologies on the IaaS Cloud.However, there is no work using these template deploymenttechnologies for automatic performance verifications of vir-tual servers because each user environment is different. Weuse Heat to provision user virtual environments by a concretetemplate and execute performance test cases automatically toshow a guarantee of performance to users.

Some tools enable automatic tests, for example, Jenk-ins [22] and Selenium [32]. However, these tools are aimedat executing automatic regression tests during the soft-ware development life cycle, and there is no tool to extractperformance test cases dynamically based on each user envi-ronment. Themethod proposed byWillmor and Embury [33]is intended to generate automatic test cases ofDB. It needs thespecifications of preconditions and post-conditions for eachDB test case. However, collecting user system specificationsis impossible for IaaS virtual machine users. Our technologycan select and execute performance tests automatically basedon installed software and connection patterns of templates.For example, it selects and executes TPC-C benchmarkwhena user system is composed of Web 3-tier model.

7 Conclusion

In this paper, we proposed a server architecture recommen-dation and automatic performance verification technology,which recommends and verifies appropriate server architec-ture on Infrastructure as a Service cloud with bare metals,containers and virtual machines. It receives an abstract tem-plate of Heat and function/performance requirements fromusers and selects appropriate servers.

Firstly, we measured UnixBench performance of a baremetal, Docker containers, KVM virtual machines controlledby OpenStack Nova to collect necessary data of appropriaterecommendation. In the results, a Docker container showedabout 75% performance compared to a bare metal, but aKVM virtual machine shows about 60% performance. Sec-

123

134 SOCA (2017) 11:121–135

ondly, we proposed a server architecture recommendationtechnology based on the measured data. It selected appropri-ate server types and created a concrete template using serverOSflexibility requirements and performance requirements ofuniform management servers. Thirdly, we proposed an auto-matic performance verification technology which executednecessary performance tests automatically on provisioneduser environments according to the template. It selecteda performance test case using information of connectionpatterns and installed software. We implemented proposedtechnologies and confirmed performance that only one houris needed for provisioning and performance verifications.Because users needed much more time to design serverarchitecture by themselves, we evaluated our proposed tech-nologies are effective to reduce users’ efforts.

In the future, we will expand target area of our proposalsuch as other IaaS platforms and PaaS layer provisioningand performance verification.We will also increase the num-ber of performance test cases for actual use cases of IaaSvirtual servers. Then, we will cooperate with IaaS Cloudservice providers to provide managed services in which ser-vice providers recommend appropriate server architectureand guarantee performance.

Acknowledgements We thank Norihiro Miura who is a manager ofthis study.

Compliance with ethical standards

Conflict of interest The author declares that he has no competing inter-ests.

Author contributions YYcarried out the proposed technology studies,designed, implemented and evaluated the proposed technology, sur-veyed the related works and drafted the manuscript. YY has read andapproved the final manuscript.

Open Access This article is distributed under the terms of the CreativeCommons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution,and reproduction in any medium, provided you give appropriate creditto the original author(s) and the source, provide a link to the CreativeCommons license, and indicate if changes were made.

References

1. Rackspace public cloud powered by OpenStack web site. http://www.rackspace.com/cloud/. Accessed 1 Feb 2016

2. OpenStack web site. http://www.openstack.org/. Accessed 1 Feb2016

3. Yamato Y, Nishizawa Y, Nagao S, Sato K (2015) Fast and reliablerestoration method of virtual resources on OpenStack. IEEE TransCloud Comput. doi:10.1109/TCC.2015.2481392

4. Barham P, Dragovic B, Fraser K, Hand S, Harris T, Ho A, Neuge-bauer R, Pratt I,WarfieldA (2003) Xen and the art of virtualization.In: Proceedings of the 19th ACM symposium on operating systemsprinciples (SOSP’03) pp 64–177

5. Kivity A, Kamay Y, Laor D, Lublin U, Liguori A (2007) KVM:the Linux virtual machine monitor. In: OLS ’07: the 2007 OttawaLinux Symposium, pp 225–230

6. Ironic web site. https://wiki.openstack.org/wiki/Ironic. Accessed1 Feb 2016

7. Docker web site. https://www.docker.com/. Accessed 1 Feb 20168. OpenStack Heat web site. https://wiki.openstack.org/wiki/Heat.

Accessed 1 Feb 20169. CloudStack web site. http://CloudStack.apache.org/. Accessed 1

Feb 201610. Amazon Elastic Compute Cloud web site. http://aws.amazon.com/

ec2. Accessed 1 Feb 201611. Pfaff B, Pettit J, Koponen T, Amidon K, Casado M, Shenker

S (2009) Extending networking into the virtualization layer. In:Proceedings of 8th ACM workshop on hot topics in networks(HotNets-VIII)

12. OpenVZ web site. http://openvz.org/Main_Page. Accessed 1 Feb2016

13. Kamp P-H, Watson RNM (2000) Jails: confining the omnipotentroot. In: Proceedings of the 2nd international SANE conference

14. Fester W, Ferreria A, Rajamony R, Rubio J (2015) An updatedperformance comparison of virtualmachines and Linux containers.In: 2015 IEEE international symposium on performance analysisof systems and software (ISPASS), pp 171–172

15. Xiao Z, SongW,ChenQ (2013)Dynamic resource allocation usingvirtual machines for cloud computing environment. IEEE TransParallel Distrib Syst 24(6):1107–1117

16. VMware vCenter Converter web site. http://www.vmware.com/products/converter. Accessed 1 Feb 2016

17. Seo KT, Hwang HS, Moon IY, Kwon OY, Kim BJ (2014) Perfor-mance comparison analysis of Linux container and virtual machinefor building cloud. Adv Sci Technol Lett 66:105–111

18. Morabito R, Kjallman J, Komu M (2015) Hypervisors vs. light-weight virtualization: a performance comparison. In: 2015 IEEEinternational conference on cloud engineering (IC2E2015) pp 386-393

19. UnixBench web site. https://code.google.com/p/byte-unixbench/.Accessed 1 Feb 2016

20. Amazon CloudFormation web site. http://aws.amazon.com/cloudformation/. Accessed 1 Feb 2016

21. Yamato Y, Muroi M, Tanaka K, Uchimura M (2014) Developmentof template management technology for easy deployment of vir-tual resources on OpenStack. J Cloud Comput 3:7. doi:10.1186/s13677-014-0007-3

22. Jenkins web site. http://jenkins-ci.org/23. Vogels W (2008) Eventually consistent. ACM Queue 6(6):14–1924. Fitzpatrick B (2004) Distributed caching with memcached. Linux

J 2004(124):525. Yamato Y (2015) Automatic verification technology of software

patches for user virtual environments on IaaS cloud. J Cloud Com-put 4:4. doi:10.1186/s13677-015-0028-6

26. TPC-C web site. http://www.tpc.org/tpcc/. Accessed 1 Feb 201627. MilojicicD, Llorente IM,MonteroRS (2011)OpenNebula: a cloud

management tool. IEEE Intern Comput 15(2):11–1428. Yamato Y, Nishizawa Y, Muroi M, Tanaka K (2015) Development

of resource management server for production IaaS services basedon OpenStack. J Inf Process 23(1):58–66

29. Yamato Y, Shigematsu N, Miura N (2014) Evaluation of agilesoftware development method for carrier cloud service platformdevelopment. IEICE Trans Inf Syst E97-D(11):2959–2962

30. Ergu D, Kou G, Peng Y, Shi Y, Shi Y (2013) The analytic hier-archy process: task scheduling and resource allocation in cloudcomputing environment. J Supercomput 64(3):835–848

31. Nathani A, Chaudhary S, Somani G (2012) Policy based resourceallocation in IaaS cloud. Future Gener Comput Syst 28(1):94–103

123

SOCA (2017) 11:121–135 135

32. Selenium web site. http://www.seleniumhq.org/. Accessed 1 Feb2016

33. Willmor D, Embury SM (2006) An intentional approach to thespecification of test cases for database applications. In: Proceedingsof the 28th international conference on software engineering, ACMpp 102–111

Yoji Yamato received his B.S., M. S. degrees in physicsand Ph.D. degrees in generalsystems studies from Univer-sity of Tokyo, Japan, in 2000,2002 and 2009, respectively.He joined NTT Corporation,Japan, in 2002. Currently he isa senior research engineer ofNTT Software Innovation Cen-ter. There, he has been engagedin developmental research ofCloud computing platform, Peer-to-Peer computing and ServiceDelivery Platform. Dr. Yamato isa member of IEEE and IEICE.

123