performance aspects of object-based storage services on ... · a network attached storage (nas) or...

c© 2017 by the authors; licensee RonPub, Lubeck, Germany. This article is an open access article distributed under the terms and conditions ofthe Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).

Open Access

Open Journal of Cloud Computing (OJCC)Volume 4, Issue 1, 2017

http://www.ronpub.com/ojccISSN 2199-1987

Performance Aspects of Object-based StorageServices on Single Board Computers

Christian Baun, Henry-Norbert Cocos, Rosa-Maria Spanou

Faculty of Computer Science and Engineering, Frankfurt University of Applied Sciences,Nibelungenplatz 1, 60318 Frankfurt am Main, Germany

[email protected], [email protected], [email protected]

ABSTRACT

When an object-based storage service is demanded and the cost for purchase and operation of servers, workstationsor personal computers is a challenge, single board computers may be an option to build an inexpensive system.This paper describes the lessons learned from deploying different private cloud storage services, which implementthe functionality and API of the Amazon Simple Storage Service on a single board computer, the development of alightweight tool to investigate the performance and an analysis of the archived measurement data. The objectiveof the performance evaluation is to get an impression, if it is possible and useful to deploy object-based storageservices on single board computers.

TYPE OF PAPER AND KEYWORDS

Regular research paper: single board computers, private cloud, object-based storage services, simple storage service

1 INTRODUCTION

A specific sort of cloud services, which belongs to theInfrastructure as a Service (IaaS) delivery model, is theobject-based storage services. Examples for public cloudofferings, which belong to this sort of services, arethe Amazon Simple Storage Service (S3) and GoogleCloud Storage, which also implements the S3-API.Furthermore exist several free private cloud solutions,which implement the S3-functionality and API.

For research projects, educational purposes likestudent projects and in environments with limitedspace and/or energy resources, single board computersmay be a useful alternative to commodity hardwareservers, because they require lesser purchase costsand operating costs. In addition, such single boardcomputer systems can be constructed in a way that theycan easily be transported by the users [2] because oftheir low weight and compact design. Single board

computer systems are not only well suited for manycloud computing applications, but also for Internet ofThings (IoT) [19] [47] scenarios and more moderndistributed systems architecture paradigms like FogComputing [7] [25] and Dew Computing [39] [45].

The drawback of single board computers is the limitedhardware resources, which cannot compete with theperformance of higher-value systems [3]. For this work,the Raspberry Pi 3 single board computer was selectedas hardware platform, because the purchase cost forthis device is just approximately 40e and operatingsystem images of several different Linux distributionsexist for its architecture. The computer provides fourCPU cores (ARM Cortex A8), 1 GB of main memoryand a 10/100 Mbit Ethernet interface. A microSD card isused as local storage.

In [2] we already investigated the characteristics,as well as the performance and energy-efficiency of amobile cluster of eight Raspberry Pi 1 Computers and

1

http://creativecommons.org/licenses/by/4.0/

http://www.ronpub.com/ojcc

Open Journal of Cloud Computing (OJCC), Volume 4, Issue 1, 2017

its single components. In this work we also mentioneduseful applications for such a system. In [3] we analyzedand compared the performance and energy-efficiencyof different clusters of Single Board Computers withthe High Performance Linpack (HPL) [20] benchmark.This work covered clusters of Raspberry Pi 1, BananaPiand Raspberry Pi 2 nodes. For this work, several freeprivate cloud S3-reimplementations have been evaluatedand deployed on a Raspberry Pi 3 when it was possible.The performance and energy efficiency have beeninvestigated among others with S3perf, a self-developedtool that investigates the required time to carry outtypical storage service operations.

This paper is organized as follows. Section 2contains a discussion of related work. In Section 3, thedeployment of different private cloud storage serviceson the Raspberry Pi 3 is described. Section 4 presentsthe development and implementation of the tool S3perf.An analysis of the gained measurement data is done inSection 5. Section 6 presents conclusions and directionsfor future work.

2 RELATED WORK

In the literature, several works cover the topic ofmeasuring the performance of storage services with theS3 interface.

Garfinkel [18] evaluated in 2007 the throughput viaHTTP GET operations which Amazon S3 can deliverwith objects of different sizes over several days fromseveral locations by using a self-written tool. Thework is focused on download operations and does notanalyze the performance of other storage service relatedoperations in detail. Unfortunately, this tool has neverbeen released by the author and the work does notinvestigate the performance and functionality of privatecloud scenarios.

Palankar et al. [32] evaluated in 2008 the abilityof Amazon S3 to provide storage support to large-scale science projects from a cost, availability, andperformance perspective. Among others, the authorsevaluated the throughput (HTTP GET operations) whichS3 can deliver in single-node and multi-node scenarios.They also measured the performance from differentremote locations. No used tool has been released by theauthors and the work does not mention the performanceand functionality of private cloud scenarios.

Zheng et al. [50] described in 2012 and 2013 the CloudObject Storage Benchmark (COSBench) [12], which isable to measure the performance of different object-based storage services. It is written in Java and providesa web-based user interface and helpful documentationfor users and developers. The tool supports not only

the S3 API, but also the Swift API and it can simulatedifferent sorts of workload. It is among others possibleto specify the number of workers, which interact withthe storage service, and the read/write ratio of the accessoperations [49]. The complexity of COSBench is alsoa drawback, because the installation and configurationrequires some effort.

McFarland [43] implemented in 2013 two Pythonscripts, which make use of the boto [8] library to measurethe download and upload data rate of the Amazon S3service offering for different file object sizes. Thosesolutions offer only little functionality. They just allowto measure the required time to execute upload anddownload operations sequentially. Parallel operationsare not supported and also fundamental operations otherthan the upload and download objects are not considered.Furthermore, the solution does not support the SwiftAPI.

Land [26] analyzed in 2015 the performance ofdifferent public cloud object-based storage services withfiles of different sizes by using the command line toolsof the service providers and by mounting buckets of theservices as file systems in user-space. This work doesnot analyze the performance and functionality of privatecloud scenarios. The author uploaded for his work afile of 100 MB in size and a local git repository withseveral small files into the evaluated storage services andafterwards erased the files, but in contrast to our work, hedid not investigate the performance of the single storageservice related operations in detail.

Bjornson [6] measured in 2015 the latency – timeto first byte (TTFB) – and the throughput of differentpublic cloud object-based storage services by using aself-written tool. Unfortunately, this tool has never beenreleased by the author. The work does not considerthe typical storage service related operations and it doesnot analyze the performance and functionality of privatecloud scenarios.

In contrast to the related works in this section, weevaluate the performance of private cloud solutionson single board computers and not the performanceof one or few public cloud service offerings. Inaddition, we developed and implemented a flexible andlightweight solution to analyze the performance of themost important storage service operations and not onlyof the HTTP GET operation or of the latency. We usedthis tool to analyze the performance of storage servicesolutions, which has also been released as free software.

2

C. Baun, H.-N. Cocos, R.-M. Spanou: Performance Aspects of Object-based Storage Services on Single Board Computers

3 DEPLOYMENT OF PRIVATE CLOUDSTORAGE SERVICES WITH THE S3-INTERFACE ON A SINGLE BOARDCOMPUTER

For this work, several free private cloud storage services,which re-implement the functionality of the Amazon S3and its API, have been taken into consideration. Table 1presents an overview of the investigated storage servicesand their characteristics.

Some of the investigated services (Cumulus, FakeS3, S3ninja, S3rver, Scality S3 Server and Walrus)support only single-node deployments, which limits thereliability and scalability of these services in principle.These services focus on providing lightweight solutions,mainly for testing and development purposes. Ceph-RGW, Minio, Riak CS and Swift are designed toimplement multi-node deployments.

The table among others highlights the programminglanguages, in which the different storage servicesolutions are implemented in. This information maybe important for application scenarios where additionalfunctions need to be implemented by the users.Furthermore, in practice, not all systems provide allcompilers or runtime environments.

When implementing a single-node service, thedurability of the stored data depends entirely on thestorage system, which is used. It is possible to use theseservices as a front end and via the S3 interface to access

• a folder inside a POSIX file system, which islocated on a local connected storage drive,

• a distributed file system, which spans via multiplephysical servers,

• a network attached storage (NAS) or even

• a storage area network (SAN).

When using a multi-node deployment with Ceph-RGW, Minio, Riak CS or Swift, the data is replicatedover several nodes by the storage service itself.

Whenever possible, the services, described in thiswork have been deployed on a Raspberry Pi 3 singleboard computer. The operating system used wasRaspbian GNU/Linux 8.0 – which is a Debian Jessiederivate – with release date 2017-03-02. The used Linuxkernel was revision 4.4.50. For this work, all deployedstorage services used a local folder on the microSDcard, which is connected to the Raspberry Pi 3, to storethe buckets and objects. In order to investigate theperformance of the services, the required time to carryout a set of operations was measured.

3.1 Ceph-RGW

The distributed file system Ceph [46] supports amongothers the Amazon S3 REST API [11]. Thisfunctionality of Ceph is also called Ceph ObjectGateway or RADOS Gateway (RGW). Ceph is freesoftware and licensed according to the GNU LesserGeneral Public License (LGPL).

Despite several success stories [1] [13] [14] [44] inliterature, which describe the deployment of Ceph onRaspberry Pi 3 computers, our attempts to install andconfigure Ceph inside the used Raspbian revision did notresult in a stable testbed. Therefore, for this work, Cephwas not further evaluated.

3.2 Cumulus

The storage service Cumulus [9] has been developed inthe context of the Nimbus infrastructure project [24].Cumulus is developed in Python and its source code islicensed according to the Apache License 2.0 [30]. TheCumulus service allows only single-node deployments.Because Cumulus does not rely on any higher levelNimbus libraries, it is possible to install it as a standalone storage service without the entire Nimbus IaaSsolution.

In order to install Cumulus, the operating systemmust provide the Python interpreter 2.5 or a morerecent revision. The single installation steps are welldocumented in the Cumulus documentation inside theQUICKSTART.txt file, which can be found inside theCumulus source code repository.

In case of a Cumulus only installation, theuser management functionality of the Nimbusinfrastructure cannot be used. For such scenarios,Cumulus provides a set of command line tools(cumulus-add-user, cumulus-list-usersand cumulus-remove-user) to create, remove, andprint out a list of current users.

After the first start of the service, all relevantconfiguration data is stored inside the file˜/cumulus/etc/cumulus.ini. Among others,the following parameters are specified: The port numberof Cumulus, the path of the folder, which is used to storethe buckets and objects, the hostname, the path and filename of the logfile and if HTTPS shall be used.

3.3 Fake S3

The Fake S3 service is developed in the programminglanguage Ruby and until revision v0.2.5, the source codeis licensed according to the MIT License [17]. Laterrevisions are not licensed according to a software licensewhich complies with the open source definition of the

3


Table 1: Storage services which implement the S3-API and their characteristics

Service Tested revision Multi-node Programming Licensedeployment language

Ceph-RGW v10.2.7 possible C++ LGPLCumulus v2.10.1 impossible Python Apache License 2.0Fake S3 v1.0.0 impossible Ruby MIT LicenseMinio v2017-03-16 possible Go Apache License 2.0Riak CS v2.1.1 possible Erlang Apache License 2.0S3ninja v2.7 impossible Java MIT LicenseS3rver v0.0.7 impossible Node.js MIT LicenseScality S3 Server v7.0.0 impossible Node.js Apache License 2.0Swift (with the swift3 middleware) v2.13.0 possible Python Apache License 2.0Walrus v4.4.0 impossible Java GPLv3

Open Source Initiative (OSI). The service offers onlysingle-node deployments and no graphical user interface.

In order to deploy Fake S3, the programming languageRuby and the package manager RubyGems need to beinstalled first. The installation of Fake S3 is done via thecommand gem install fakes3. Important servicerelated parameters like the port number and the pathof the S3 data folder are provided as command lineparameters when starting the Fake S3 service.

During the evaluation of Fake S3, some bugs havebeen observed with this service. Buckets are still seenby the service, even if they have been erased prior. Withthe evaluated revisions, it was not possible to modify theaccess key and the secret access key via command lineparameters or to specify them via environment variables.

3.4 Minio

The Minio service is developed in the programminglanguage Go and its source code is licensed accordingto the Apache License 2.0 [28]. Minio provides a webuser interface, which allows to carry out all object andbucket related tasks. The Minio project offers not onlythe source code of the service, but also binaries forthe operating systems Mac OS X, Linux, FreeBSD andMicrosoft Windows. Linux binaries for x86-compatiblearchitectures and even ARM architectures are available.

When starting the service, the port number and a localfolder for the object data, are provided as commandline parameters. The user access key and secret accesskey can be specified via the environment variablesMINIO_ACCESS_KEY and MINIO_SECRET_KEY.

During the first start of the service, allrelevant configuration data is stored inside the file˜/.minio/config.json. Inside this file, amongothers the following parameters are specified: Theuser access key and secret access key, if the web userinterface shall be used, if the server shall print out

messages on command line and the logging level and ifthe server shall write messages in a logfile and the filename as well as the logging level.

Since November 2016, Minio provides multi-disksupport with internal replication on single nodedeployments and across multiple nodes. To avoid asingle node being a bottleneck for requests, a tool likethe light-weight web server software nginx can be usedas a proxy [27].

3.5 Riak CS

Riak CS (Cloud Storage) implements a S3-like objectstorage with the S3-API on top of Riak KV [29], whichis a distributed NoSQL key-value data store. Riak KVimplements the characteristics of Amazon Dynamo [15]and has therefore a focus on multi-node scalability,fault-tolerance and data durability thanks to internalreplication.

The evenly distribution of stored data is carried out byRiak KV fully automatic. To achieve this characteristic,Riak KV places the keys inside a Distributed Hash Table(DHT) [23]. The DHT is split into partitions, circularlydistributed among the nodes, and replicas are alwaysplaced in contiguous partitions [10].

To enforce the global uniqueness of entities, a Riak CSstorage system also requires the service Stanchion [41],which serializes the requests that involve creation andmodification of the entities. When a user for instancetries to create a bucket with an already existing bucketname, this request is rejected by Stanchion. The sameresult is caused by the attempt to create a user with theEmail address of an already existing user [22].

Riak KV, Stanchion and Riak CS are implemented inthe programming language Erlang and are free software.The source code [34] is licensed according to the ApacheLicense 2.0. The installation of Riak KV, Stanchionand Riak CS, as well as a compatible revision of the

4


Erlang compiler, requires a lot of effort, because no pre-compiled packages were available for Raspbian and theARM architecture of the Raspberry Pi. Compiling allrequired components requires more than an hour on aRaspberry Pi 3. The services Riak KV, Stanchion andRiak CS have their own configuration files and need tobe started in the correct order.

With Riak CS Control, a standalone web applicationfor the user management of Riak CS systems exists.Equal to the other Riak components already described,Riak CS Control is free software and implemented inErlang. Unfortunately, all attempts to deploy Riak CSControl on the Raspberry Pi 3 failed in this work.

3.6 S3ninja

The S3 ninja service is developed in the programminglanguage Java and its source code is licensed accordingto the MIT License [36]. The service offers only single-node deployments. S3ninja provides a web user interfacewhich allows to carry out all object and bucket relatedtasks.

From a perspective of operating system, S3ninjarequires just an installed Java runtime environment. Ineffect, the deployment and start of this service solutionrequire little effort. An unexpected issue arose from theinteraction of S3ninja with s3cmd: The used s3cmdrevision v2.7 forces the API to be accessible at path /,but S3ninja only permits the path /s3. This issue wassolved by installing nginx and using it as a proxy.

In the configuration file of the service˜/s3ninja/app/application.conf, theuser access key and the secret access key can bespecified. Also the path of the S3 data folder and theport number is specified inside this file.

3.7 S3rver

The service S3rver is developed in the programminglanguage JavaScript and its source code is licensedaccording to the MIT License [37]. The software isexecuted on server-side by using Node.js. The serviceoffers only single-node deployments and no graphicaluser interface.

In order to deploy S3rver, the Node.js run-timeenvironment and the package manager npm need to beinstalled first. The installation of S3srver is done via thecommand npm install s3rver -g. Importantservice related parameters like the port number and thepath of the S3 data folder are provided as commandline parameters when starting the service. An unusualcharacteristic of S3rver is, that the IP address also needsto be specified as a command line parameter. If not

specified, the port number of the service only acceptsconnections from localhost.

With the evaluated revisions, it was not possibleto modify the access key and the secret access keyvia command line parameter or to specify them viaenvironment variables.

3.8 Scality S3 Server

The Scality S3 Server, which offers only single-nodedeployments, is developed in the programming languageJavaScript and its source code is licensed according tothe Apache License 2.0 [38]. The software is executedon server-side by using the Node.js JavaScript runtimeenvironment. If Node.js and the package manager npmare already installed, the service can be deployed via thecommand npm install.

One characteristic of Scality S3 Server is that itsmain memory consumption exceeds the physical mainmemory of the Raspberry Pi 3. In order to solve thisissue, a Swap partition of at least 1 GB of size or a Swapfile is required.

The user access key and secret accesskey can be specified via the environmentvariables SCALITY_ACCESS_KEY_ID andSCALITY_SECRET_ACCESS_KEY or inside thefile ˜/S3/conf/authdata.json. Further relevantconfiguration parameters like the logging level and theport number of the service are specified inside the file˜/S3/config.json.

3.9 Swift

Swift is a component of the IaaS solution OpenStack.It is written in Python and licensed according to theApache License 2.0 [31]. The OpenStack projectwas initiated in 2010 by Rackspace and NASA withthe objective to develop a scalable, durable and high-available object storage, which can be deployed acrossa large number of nodes. The swift service ensures datareplication and integrity across the connected nodes.

Although being an object-storage service like theother services examined in this work, Swift implementsan API, which is different to the S3-API. For interactionon command line, the python client [33] for the SwiftAPI is a working solution. If the S3-API is mandatory, amiddleware like swift3 [42] needs to be deployed.

The Swift service offers two options forauthentication. The first option requires the OpenStackKeystone service installed and the second optionmakes use of the TempAuth functionality, which isimplemented in Swift. The Keystone option wasdiscarded, because a lightweight solution was wanted.The user credentials when using TempAuth are specified

5


inside the file /etc/swift/proxy-server.conf.Further relevant configuration parameters, which arealso specified inside this file, are among others thelogging level and the port number of the swift service.

The deployment of Swift on the Raspberry Pi 3needed much effort because of the several softwaredependencies and the complex configuration. Theinstallation of the latest revision 2.14.0 was possible, butbecause of a software bug or a configuration issue, it wasonly possible to create buckets, but impossible to uploadobjects into the service. Therefore, revision 2.2.0 ofthe Swift service was installed, because packages of thisrevision for the Raspbian operating system do alreadyexist.

3.10 Walrus

Walrus is a component of the IaaS solutionEucalyptus [16]. It is written in Java and licensedaccording to the GPLv3 license. The development ismainly driven by Hewlett Packard Enterprise (HPE),since HP acquired the company Eucalyptus Systems in2014. Of all private cloud service solutions, mentionedin this work, Walrus has the longest development period.It is a part of Eucalyptus since revision 1.4 in 2009.

The developers support only the installation ofEucalyptus inside the operating systems CentOS 7and Red Hat Enterprise Linux 7. Installations ondifferent Linux distributions cause much effort becauseof multiple dependencies. Walrus uses a folder insidea POSIX file system to store the buckets as folders andthe object data as files. The durability of the stored datadepends entirely on the used storage system.

A stand alone installation of Walrus out of the box isimpossible because Walrus does not utilize the POSIXfile system for storing the metadata. These are stored inthe database, which is managed by the Cloud Controller(CLC) – a different Eucalyptus component.

The service allows only single-node deployments.Instead of adding multi-node functionality into Walrus,the developers decided to support using Riak CS orCeph-RGW as object storage in Eucalyptus instead ofWalrus.

A stand-alone installation of Walrus is impossibleand the installation of Eucalyptus was not successful onRaspbian GNU/Linux. Installation attempts with Ubuntu16.04 LTS failed the same. Therefore, for this work,Walrus was not further evaluated.

4 DEVELOPMENT OF S3PERF

An analysis of the existing testing solutions (seesection 2) resulted in the development of a new tool,called S3perf [5]. Its focus was to be lightweight, be

Table 2: Description of the HTTP methods withrequest-URIs that are used to interact with storageservices

Account-related OperationsGET / List bucketsHEAD / Retrieve metadata

Bucket-related OperationsGET /bucket List objectsPUT /bucket Create bucketDELETE /bucket Delete bucketHEAD /bucket Retrieve metadata

Object-related OperationsGET /bucket/object Retrieve objectPUT /bucket/object Upload objectDELETE /bucket/object Delete objectHEAD /bucket/object Retrieve metadataPOST /bucket/object Update object

compatible with the S3-API and the Swift API, haveonly few dependencies, cause only little effort for theinstallation and be simple to use.

One well-documented option to develop software,which shall interact with AWS-compatible services, isusing the programming language Python together withthe boto [8] library. In order to even simplify theimplementation, the command line tools s3cmd [35]and the Swift client [33] were selected to carry outall interaction with the used storage services. Thesepreconditions led to the development of a bash script.

Once a storage service is registered in theconfiguration file of s3cmd, S3perf can interact withits S3-compatible REST API. REST is an architectural-style that relies on HTTP methods like GET or PUT.S3-compatible services use GET to receive the list ofbuckets that are assigned to an user account, or a listof objects inside a bucket or an object itself. Bucketsand objects are created with PUT and DELETE is usedto erase buckets and objects. POST can be used toupload objects and HEAD is used to retrieve meta-datafrom an account, bucket or object. Uploading files intoS3-compatible services is done via POST directly fromthe client of the user. Table 2 gives an overview ofmethods used to interact with S3-compatible storageservices. [4]

If the Swift API shall be used for the communicationwith a storage service, S3perf does not use the commandline tool s3cmd, but the Swift client. In contrast tos3cmd, the swift tool uses no configuration file tostore user credentials. The user of S3perf needs to

6


0.01

0.1

1

10

100

1 10 100 1000 10000 100000 1x106 1x107

Thro

ug

hp

ut

[Mb

its/

s]

Message (payload) size [Bytes]

NetPIPE Bandwidth (logarithmic scaling of the x-axis and y-axis)

NetPIPE using the end-to-end protocol TCP 0.0001

0.001

0.01

0.1

1

1 10 100 1000 10000 100000 1x106 1x107

Late

ncy

[s]


NetPIPE Latencies (logarithmic scaling of the x-axis and y-axis)

0

20

40

60

80

100

1 10 100 1000 10000 100000 1x106 1x107

Thro

ug

hp

ut

[Mb

its/

s]


NetPIPE Bandwidth (logarithmic scaling of the x-axis)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1 10 100 1000 10000 100000 1x106 1x107

Late

ncy

[s]


NetPIPE Latencies (logarithmic scaling of the x-axis)

Figure 1: Analysis of the network performance of the Raspberry Pi 3 by using the NetPIPE benchmark

specify the endpoint of the used storage service, aswell as username and password inside the environmentvariables ST_AUTH, ST_USER and ST_KEY.

A common assumption, when using storage services,which implement the same API, is that all operationscause identical results. But during the development ofS3perf, some challenges caused by non-matching servicebehavior emerged. One issue is the encoding of thebucket names. In order to be conforming to the DNSrequirements, bucket names should not contain capitalletters. To comply with this rule, the services Minio,Riak CS, S3rver and Scality S3 do not accept bucketnames with capital letters. Other services like NimbusCumulus and S3ninja only accept bucket names, whichare encoded entirely in capital letters. The serviceofferings Amazon S3 and Google Cloud, as well as theprivate cloud solutions Fake S3 and OpenStack Swift aremore generous in this case and accept bucket names,which are written in lowercase and capital letters. Inorder to be compatible with the different storage servicesolutions and offerings, S3perf allows the user to specifythe encoding of the bucket names.

When doing a performance evaluation with S3perf,the tool executes these six steps for a specific numberof objects of a specific size:

1. create a bucket

2. Upload one or more objects into this bucket

3. Fetch the list of objects inside the bucket

4. Download the objects from the bucket

5. Erase the objects inside the bucket

6. Erase the bucket

The time, which is required to carry out theseoperations, is individually measured and can be usedto analyze the performance of these commonly usedstorage service operations.

Users of S3perf have the freedom to specify thenumber of files via command-line parameters, whichshall be created, uploaded and downloaded, as well astheir individual size. The files are created via the tool ddand contain pseudorandom data from /dev/random.

In order to be able to simulate different load scenarios,the S3perf tool supports the parallel transfer of objects,as well as requesting delete operations in parallel. If theparallel flag is set, the steps 2, 4 and 5 are executed inparallel by using the command line tool parallel.

Every time when S3perf is executed, the tool printsout a line of data, which informs the user about the date

7


and time when the execution was finished, the number ofcreated objects, the size of the single objects, as well asthe required time in seconds to execute the steps 1-6. Thefinal column contains the sum of all time values, whichis calculated by using the command line tool bc.

The structure of the output simplifies the analysis ofthe performance measurements by using tools like sed,awk and gnuplot.

5 PERFORMANCE ANALYSIS OF THEDEPLOYED CLOUD STORAGE SERVICES

To generate a sufficient number of measurement data,S3perf was executed for each storage service five timeswith ten objects of sizes 512 Byte, 1 kB, 2 kB, 4 kB,8 kB, 16 kB, 32 kB, 64 kB, 128 kB, 256 kB, 512 kB,1 MB, 2 MB, 4 kB and 8 MB. This caused up to 75test runs for each storage service. Few test runs failedbecause the resources of the used single board computerwere fully utilized or because the service did not allowto upload files of a specific size.

When analyzing the required time to upload ordownload files, it must be kept in mind that theRaspberry Pi 3 is equipped with a 10/100 Mbit Ethernetcontroller, that is internally connected with the USBhub. In practice, the network performance is lessthan 100 Mbit. The network performance between twoRaspberry Pi 3 nodes was measured with the command-line tool iperf v2.0.5 [21] and with the NetPIPE v3.7.2benchmark [40]. According to iperf, the networkperformance between the two nodes is approximately94 Mbit per second.

A more detailed analysis of the network performanceis possible with the NetPIPE benchmark. It tests thelatency and throughput over a range of message sizesbetween two processes. The benchmark was executedbetween the two nodes by just using TCP as end-to-end(transport layer) protocol. The results in Figure 1 showthat the smaller a message is, the more dominant thetransfer time by the communication layer overhead is.For larger messages, the communication rate becomesbandwidth limited by a component in the communicationsubsystem. Examples are the data rate of the networklink, the utilization of the transmission medium or aspecific device between the sender and the receiver likethe used network switch.

5.1 Step 1: Create a Bucket

The creation of a single bucket (Step 1) cannot beexecuted in parallel and it is not influenced by thesize of the objects, which are (in later steps) createdand transferred by S3perf. Therefore, the boxplot inFigure 2, which presents the statistical distribution of the

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Cum

ulu

s

Fake

S3

Min

io

Ria

k C

S

S3

nin

ja

S3

rver

Sca

lity

Sw

ift

[s]

Required Time to create a Bucket (50 Tests)

Figure 2: Time to create a bucket inside the testedstorage services

required time [s], does not mention any object sizes. Thehorizontal line inside each box is the median (secondquartile). The smallest 50% of the points are smallerthan or equal to the median value. 25% of the points aresmaller than or equal to the bottom box boundary (firstquartile) and 25% of the points are bigger than or equalto the top box boundary (third quartile).

The size of the box is also called interquartile range(IQR). It is calculated via (IQR = Q3 − Q1) and usedto find outliers in the data. The whiskers extend fromthe ends of the box to the most distant point whose y-axis value has a maximum distant of 1.5 times the IQR.Figure 2 does not show all outliers. In few cases, creatinga bucket with Riak CS and Swift required around 5seconds.

Figure 2 shows that Riak CS and Swift require moretime to create a bucket compared with the other servicesolutions. This is probably caused by their more complexway to store the data with replicas, which consumesadditional resources, but in principle also offers the

8


option to build systems with a better level of reliability.

5.2 Step 2: Upload the Objects into the Bucket

The required time to upload ten objects sequentially andin parallel into the tested storage services (Step 2) ispresented in Figure 3. The measurements show that theparallel upload of the ten objects only in few cases lead toa significant acceleration compared with the sequentialupload. In most cases, the period to upload the objectsin a parallel way is longer compared to the sequentialupload. The parallel upload is significant beneficialwhen using Fake S3 with objects > 512 kB, Minio withobjects > 1024 kB and < 8192 kB, Riak CS with objects> 256 kB and S3ninja with objects > 32 kB.

From the evaluated services, Riak CS is the most mainmemory consuming one and it does not free the mainmemory after an upload operation. After the uploadof ten objects in parallel with a size of 2048 kB each,only approximately 170 MB of main memory have beenavailable on the Raspberry Pi 3. The attempt to uploadten objects in parallel with a size of 4096 kB crashedthe node. When trying to upload ten objects in serialwith a size of 8192 kB, the main memory of the nodegets entirely filled and also the Swap storage is used. Ineffect, the node also crashed during this attempt.

The upload of objects > 1024 kB into S3ninja failedbecause nginx, which was used as a proxy, didnot permit the upload of such large files. Anotherobservation with S3ninja was, that the service consumedone CPU core entirely at times, during the upload of tenobjects (in parallel and sequential operation mode).

Also Swift consumes a lot of main memory becauseof the way, the objects, buckets and accounts are storedin rings with replicas. When deploying Swift on serverresources, the replicas are distributed over multiplephysical storage devices. On the Raspberry Pi 3 node,all replicas are stored on the microSD card, which is alsoused to store the operating system.

5.3 Step 3: Fetch the List of Objects

Fetching a list of objects (Step 3) cannot be executedin parallel and should not actually depend on the sizeof the objects, but as presented in Figure 4, when usingS3ninja, the size of the objects has a strong influence onthe required time. The root cause for this behavior maybe the working method of the Java VM used.

It is also observed in this step that Riak CS and Swiftprovide a lesser performance compared with the otherinvestigated solutions (except S3ninja) because Riak CSand Swift both store the objects with replicas.

5.4 Step 4: Download the Objects

The required time to download ten objects sequentiallyand in parallel into the tested storage services (Step 4)is presented in Figure 5. The measurements show thatthe parallel download of the ten objects does only in fewcases lead to a significant acceleration compared with thesequential upload. In most cases, the period to downloadthe objects in a parallel way is longer compared to thesequential download. The parallel upload is significantbeneficial when using Cumulus with objects > 1 kB and< 2048 kB and S3ninja with objects > 16 kB.

5.5 Step 5: Erase the Objects

The required time to erase ten objects sequentially andin parallel with the tested storage services (Step 5) ispresented in Figure 6. The measurements show thaterasing the ten objects in parallel does only in fewcases lead to a significant acceleration compared withthe sequential upload. In most cases, the period toerase the objects in a parallel way is longer comparedto the sequential operation mode. Erasing in parallel isbeneficial when using S3ninja with objects > 8 kB.

Erasing objects should not actually depend on the sizeof the objects, but as presented in Figure 6, when usingS3ninja, the size of the objects has a strong influence onthe required time. The root cause for this observationmay be the working method of the Java VM used.

5.6 Step 6: Erase the Bucket

Erasing a single bucket (Step 6) cannot be executed inparallel and because the bucket is already empty, theperformance of this operation is not be influenced by thesize of any objects. Therefore, the boxplot in Figure 7,which presents the statistical distribution of the requiredtime [s], does not mention any object sizes.

The Figure does not show all outliers. In few cases,erasing a bucket with Riak CS required around 8 secondsand with Swift around 5 seconds.

6 CONCLUSIONS AND NEXT STEPS

The performance of single board computers like theRaspberry Pi 3 cannot compete with higher-valuesystems because the characteristics of their relevanthardware components, especially the CPU, mainmemory, storage and network interface. Regardless ofthe performance, single board computers are useful foracademic purposes and research projects because of thelesser purchase costs and operating costs compared withcommodity hardware server resources. The hardwareresources of a single Raspberry PI 3 node are sufficientto deploy and use each one of the S3-API-compatible

9


0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]

Time to upload 10 Objectsin parallel (Mean Time of 5 Tests)

1.1

88

21

.22

28

1.2

03

21

.16

18

1.2

91

81

.21

84

1.2

05

21

.22

98

1.5

41

.42

18

1.6

38

62

.22

88

3.1

05

64

.96

22

9.1

94

6

Cumulus

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]

Time to upload 10 Objectssequentially (Mean Time of 5 Tests)

0.8

85

0.8

86

60

.76

76

0.7

14

20

.69

88

0.7

06

60

.71

92

0.8

39

0.8

22

21

.03

68

1.3

40

41

.82

36

2.7

55

64

.68

78

.52

84Cumulus

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.0

60

61

.03

74

1.0

58

41

.04

08

1.0

46

61

.08

91

.07

86

1.0

93

21

.15

46

1.1

94

61

.39

36

1.9

06

82

.71

56

4.4

94

88

.48

42Fake S3

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.1

58

81

.36

56

0.5

47

80

.84

36

0.6

25

20

.60

22

1.5

14

21

.00

41

.21

32

1.9

12

42

.45

06

4.5

85

88

.93

761

8.6

55

83

4.1

19

2Fake S3

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.0

05

80

.99

46

1.0

13

40

.98

11

.00

94

1.1

62

61

.18

41

.15

86

1.1

06

81

.49

62

2.9

47

2.5

68

82

.68

16

4.4

35

42

0.5

82

4Minio

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.7

24

20

.70

40

.92

14

0.4

03

20

.69

88

0.4

34

20

.66

46

1.1

29

20

.65

16

1.2

89

1.4

58

2 2.7

60

84

.46

44

10

.17

34

18

.56

34Minio

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Node c

rash

ed

Node c

rash

ed

Node c

rash

ed

[s]

Object Size [kB]


3.0

70

61

.35

28

1.3

67

81

.34

98

1.3

81

21

.95

76

1.9

87

6 3.2

63

85

.46

54

1.8

89

66

.95

68

5.7

32

6

Riak CS

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Node c

rash

ed

[s]

Object Size [kB]


4.8

69

81

.92

56

3.6

99

41

.58

24

1.3

51

81

.95

76

3.7

32

2.0

10

2 3.1

62

9.3

39

46

.48

12

9.5

65

83

9.3

65

68

1.5

17

2Riak CS

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Opera

tion f

aile

dO

pera

tion f

aile

dO

pera

tion f

aile

d

[s]

Object Size [kB]


1.8

45

61

.21

62

1.2

10

21

.23

88

1.2

92

41

.45

36

1.5

76

82

.04

48

2.8

18

4.4

62

26

.75

88 1

2.6

16

6S3ninja

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Opera

tion f

aile

dO

pera

tion f

aile

dO

pera

tion f

aile

d

[s]

Object Size [kB]


1.4

30

.87

04

0.8

45

40

.89

34

0.9

29

61

.25

46

2.0

31

42

.70

42

4.4

66

8.8

09

61

5.9

66

26

.86

08S3ninja

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.0

02

21

.00

04

0.9

81

60

.97

32

1.0

02

81

.03

52

0.9

98

0.9

81

81

.08

82

1.2

25

1.4

06

1.9

43

42

.57

54

.30

02

8.4

67

8S3rver

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.5

81

0.5

77

80

.28

86

0.2

78

0.2

71

60

.30

80

.30

52

0.3

46

20

.40

52

0.5

27

40

.84

22

1.4

33

4 2.5

61

44

.83

64

9.3

70

8

S3rver

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


2.1

20

61

.40

44

1.3

61

.68

08

1.4

02

81

.28

02

1.3

41

61

.52

94

2.0

51

82

.07

74

2.3

23

43

.07

52

7.0

28

6 10

.60

12

16

.31

42Scality

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.7

43

21

.26

74

1.0

86

1.3

05

61

.35

91

.05

66

1.1

12

21

.07

62

1.2

73

81

.46

36

1.7

79

24

.19

87

.80

74

10

.82

32

17

.37

2

Scality

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Opera

tion f

aile

d

[s]

Object Size [kB]


6.7

00

66

.31

75

.59

16 6.8

40

25

.51

25

.94

68 7

.32

98

6.6

80

86

.69

06 8

.02

87

.17

7 11

.76

71

6.8

91

25

.08

16Swift

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


3.0

23

42

.48

2.4

34

82

.45

16

2.4

91

42

.43

78

2.6

31

82

.46

46

2.6

37

24

.20

34

3.1

78

85

.54

26

9.0

44

19

.53

12

39

.81

06Swift

Figure 3: Time to upload ten objects sequentially and in parallel into the tested storage services

10


0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]

Time to fetch a list of 10 Objects(Mean Time of 5 Tests)

0.2

14

0.2

06

60

.21

74

0.2

14

60

.20

58

0.2

05

60

.20

70

.22

58

0.2

09

0.2

24

60

.23

0.2

29

0.2

06

60

.21

86

0.2

23

Cumulus

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.2

23

40

.21

48

0.2

18

20

.20

70

.21

54

0.2

13

80

.21

22

0.2

53

80

.21

76

0.2

61

20

.22

18

0.2

15

20

.21

20

.22

38

0.2

08

Fake S3

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.1

77

60

.17

56

0.1

80

40

.17

74

0.1

76

0.1

77

0.1

81

80

.18

52

0.1

77

60

.17

74

0.1

83

80

.18

40

.18

20

.18

0.1

75

2

Minio

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

[s]

Object Size [kB]


0.3

90

40

.31

54

0.3

14

20

.31

40

.31

28

0.3

60

20

.30

64

0.3

01

60

.30

74

0.3

05

80

.30

28

3.2

19

20

.35

12 1

.71

1

Riak CS

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

No O

bje

ct lis

t to

fetc

h b

eca

use

Uplo

ad f

aile

dN

o O

bje

ct lis

t to

fetc

h b

eca

use

Uplo

ad f

aile

dN

o O

bje

ct lis

t to

fetc

h b

eca

use

Uplo

ad f

aile

d

[s]

Object Size [kB]


0.5

36

60

.31

96

0.3

29

60

.42

04

0.4

93

60

.77

34

1.1

26

81

.87

02 2.9

55

25

.76

14

11

.52

72

3.3

37

S3ninja

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.2

14

60

.20

62

0.1

97

20

.19

24

0.1

95

60

.18

74

0.1

96

0.1

95

60

.22

26

0.2

06

40

.20

36

0.2

05

0.2

31

80

.22

02

0.2

53

8S3rver

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.6

34

40

.35

02

0.3

02

20

.26

96

0.2

66

80

.25

84

0.2

72

80

.26

52

0.2

42

60

.24

94

0.2

46

40

.25

32

0.2

39

0.2

46

40

.24

42

Scality

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.9

25 1.8

75

60

.65

68

0.6

78

40

.66

44

0.6

51

80

.67

92

0.6

66

2 1.9

32

40

.67

96

1.2

18

20

.68

46

5.1

18

41

.93

92

0.6

80

2

Swift

Figure 4: Time to fetch the list of objects (more detailed view)

storage service solutions, which have been examined inthis work.

In order to investigate the performance of the analyzedstorage services, a set of typical operations has beenpicked out and a lightweight tool, called S3perf has beendeveloped and implemented. This tool was used to carryout the set of operations and measure the required time.Based on these measurements, the performance of thestorage services on a single board computer has beenanalyzed.

Results are among others that the service solutionsCumulus, Fake S3, Minio, S3ninja, S3rver and Scalityfor most operations provide a better performancecompared with Riak CS and Swift. The reason forthis observation is probably caused by the way RiakCS and Swift store the objects and bucket data withreplicas and organize them via Distributed Hash Tables.Riak CS and Swift both focus to work efficientlyinside multi-node deployments and are not in first placedesigned to provide a strong performance inside singlenode scenarios. Another observation was that theparallel execution of multiple upload, download anderase operations just in few scenarios are beneficialfor the overall performance. Especially, when using

objects of 1 MB and more, the maximum throughput ofthe network interface of the Raspberry Pi 3 becomes alimiting factor. Furthermore, it has been observed thatwhen using S3ninja, the performance of list and deleteoperations depend on the size of the objects.

Next steps are the deployment of Ceph-RGW,Minio, Riak CS and Swift inside multi-node systemsof Raspberry Pi 3 nodes and an analysis of theirperformance and robustness.

For future work, it is also useful to analyze the totalcost of ownership (TCO) of multi-node systems of singleboard computers which host storage services. As provenamong others by Zhao et al. [48], the TCO of singleboard computer clusters can be better compared withhigher-value systems.

Next steps also comprise an analysis of the maximumworkload and storage capacity of the described systemsin order to archive more information about the practicalusability of storage services which are deployed onsingle board computer clusters.

11


0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]

Time to download 10 Objectsin parallel (Mean Time of 5 Tests)

1.1

25

81

.12

72

1.0

62

21

.03

88

1.0

86

61

.10

96

1.1

19

21

.21

38

1.3

04

61

.32

51

.72

28

2.1

85

83

.08

48

4.9

58

.91

1Cumulus

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]

Time to download 10 Objectssequentially (Mean Time of 5 Tests)

0.4

18

60

.44

98

5.7

45 9

.82

92

9.7

89

49

.51

52

9.4

55

69

.53

12

7.2

73

25

.95

16 9.7

01

25

.57

62

3.0

84

5.9

14

48

.97

04Cumulus

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.0

42

41

.05

64

1.0

73

21

.03

21

.06

16

1.0

38

1.0

58

61

.05

92

1.1

25

1.1

87

21

.34

18

1.7

52

82

.50

06

4.4

03

67

.78

72

Fake S3

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.9

27

20

.87

48

0.4

82

20

.47

60

.48

64

0.4

78

80

.52

18

1.0

78

40

.75

58

1.3

71

1.4

21

62

.34

52

4.6

09

47

.71

72

16

.30

78Fake S3

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.9

59

40

.94

92

0.9

65

40

.94

96

0.9

35

80

.96

26

0.9

61

80

.95

91

.01

24

1.1

14

1.2

78

21

.73

48

2.6

04

84

.45

22

7.9

76

Minio

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.2

54

20

.26

38

0.2

70

40

.25

96

0.2

64

60

.27

20

.26

0.2

90

20

.35

48

0.4

79

0.7

20

61

.2 2.1

09

83

.89

77

.48

64

Minio

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

[s]

Object Size [kB]


1.1

33

41

.08

14

1.0

72

61

.07

52

1.0

81

41

.14

11

.14

81

.16

08

1.2

11

21

.35

28

1.5

61

23

.18

88

Riak CS

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

[s]

Object Size [kB]


0.7

54

40

.71

78

0.7

92

60

.74

20

.72

84

0.7

69

0.7

78

20

.82

02

0.8

75

21

.03

48

1.4

40

23

.30

42 4

.68

82

8.3

22

6Riak CS

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

No O

bje

ct t

o d

ow

nlo

ad b

eca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o d

ow

nlo

ad b

eca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o d

ow

nlo

ad b

eca

use

Uplo

ad f

aile

d

[s]

Object Size [kB]


1.2

49

21

.20

98

1.2

39

21

.24

41

.26

11

.56

68

1.6

75

1.9

90

42

.46

48 3

.91

64

6.6

28

6 11

.69

S3ninja

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

No O

bje

ct t

o d

ow

nlo

ad b

eca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o d

ow

nlo

ad b

eca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o d

ow

nlo

ad b

eca

use

Uplo

ad f

aile

d

[s]

Object Size [kB]


0.9

96

41

.14

44

1.2

07

41

.32

86

1.3

09

61

.75

283

.31

22

4.3

06

67

.88

92

14

.92

92

7.0

32

45

2.3

59

8S3ninja

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.0

46

0.9

96

20

.96

66

0.9

63

40

.96

78

0.9

49

60

.99

76

1.0

08

81

.05

42

1.1

32

21

.33

14

1.7

51

42

.71

08

4.7

35

47

.98

92

S3rver

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.7

19

40

.65

84

0.2

80

80

.27

98

0.2

93

60

.27

64

0.2

73

60

.30

86

0.3

72

60

.53

96

0.7

48

1.2

01

22

.06

54

3.9

05

67

.44

96

S3rver

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.2

54

81

.11

28

1.1

05

61

.07

16

1.0

59

1.0

49

21

.09

51

.10

96

1.0

96

41

.18

58

1.3

84

21

.68

34

2.4

61

24

.17

34

7.7

95

Scality

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.0

67

20

.90

42

0.6

70

80

.66

34

0.6

46

60

.60

70

.59

52

0.5

39

80

.59

46

0.6

93

60

.90

58

1.4

48

62

.39

78

4.2

47

47

.99

26

Scality

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

No O

bje

ct t

o d

ow

nlo

ad b

eca

use

Uplo

ad f

aile

d

[s]

Object Size [kB]


3.9

11

83

.30

94

3.3

52

63

.42

72

3.3

01

63

.45

22

3.4

74

23

.43

3.3

55

23

.48

26

3.7

71

84

.08

92

4.7

91

6 5.9

22

4

Swift

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.9

01

40

.93

28

0.9

31

80

.90

.91

22

0.8

87

0.9

16

0.9

35

1.0

04

21

.12

76

1.3

78

61

.79

6 2.7

21

4.4

92

88

.08

96

Swift

Figure 5: Time to download ten objects sequentially and in parallel into the tested storage services

12


0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]

Time to erase 10 Objectsin parallel (Mean Time of 5 Tests)

1.2

32

81

.11

92

1.0

96

81

.12

98

1.1

92

41

.28

71

.18

42

1.1

48

1.1

65

41

.12

72

1.2

75

61

.49

1.3

58

1.5

15

61

.14

56

Cumulus

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]

Time to erase 10 Objectssequentially (Mean Time of 5 Tests)

0.6

82

80

.94

68

0.7

26

80

.69

76

0.6

94

40

.78

78

0.7

39

80

.75

58

0.6

79

20

.68

40

.81

84

0.8

55

40

.83

26

0.7

42

60

.78

14

Cumulus

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.9

80

60

.98

32

0.9

81

20

.97

78

0.9

73

40

.97

26

0.9

72

20

.97

02

0.9

90

.97

02

0.9

75

81

.52

52

1.5

30

21

.54

86

1.0

36

Fake S3

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.4

27

0.4

01

80

.40

04

0.4

38

0.5

47

40

.39

82

0.3

84

40

.41

54

0.4

11

60

.44

86

0.7

69

20

.47

52

0.5

97

20

.44

18

0.4

86

Fake S3

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.9

49

0.9

43

60

.94

50

.94

78

0.9

49

80

.96

18

0.9

63

80

.96

50

.94

38

0.9

45

81

.77

90

.93

92

0.9

61

.12

16

0.9

82

Minio

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.2

45

20

.23

84

0.2

34

80

.23

84

0.2

33

40

.23

38

0.2

30

20

.23

56

0.2

32

20

.54

66

0.2

53

80

.24

40

.64

76

0.3

06

43

.02

72

Minio

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

[s]

Object Size [kB]


1.3

26

41

.52

98

1.6

49

21

.33

42

.08

86

2.4

11

61

.74

12

.40

02

2.2

17

11

.75

16

4.6

72

81

0.3

03

Riak CS

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

Node c

rash

ed d

uri

ng u

plo

ad A

ttem

pt

[s]

Object Size [kB]


1.6

46

41

.92

22

.51

34

2.7

66

21

.57

08 2.7

49

61

.55

82

.06

08

2.0

33

4 3.4

61

26

.34

22

10

.01

18

5.1

51

43

.35

4

Riak CS

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

No O

bje

ct t

o e

rase

beca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o e

rase

beca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o e

rase

beca

use

Uplo

ad f

aile

d

[s]

Object Size [kB]


1.1

02

41

.11

28

1.1

40

61

.09

64

1.1

01

1.1

03

61

.11

81

.08

78

1.2

53

61

.11

11

.33

22

1.1

09

2

S3ninja

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

No O

bje

ct t

o e

rase

beca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o e

rase

beca

use

Uplo

ad f

aile

dN

o O

bje

ct t

o e

rase

beca

use

Uplo

ad f

aile

d

[s]

Object Size [kB]


0.6

39

0.6

69

0.7

32

20

.81

02

1.0

71

61

.47

76

1.5

72

22

.24

48

3.8

22

7.0

53

8 12

.51

74

24

.17

58S3ninja

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.9

69

80

.95

72

0.9

78

0.9

46

0.9

37

20

.94

28

1.0

02

60

.95

54

0.9

63

40

.95

18

0.9

63

60

.94

82

0.9

63

0.9

54

0.9

95

2

S3rver

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


0.2

69

0.2

64

60

.24

48

0.2

79

0.2

52

0.2

47

40

.24

82

0.2

49

40

.25

54

0.2

70

60

.27

86

0.2

90

.36

72

0.3

67

0.4

21

S3rver

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.3

71

61

.24

28

1.2

03

1.3

29

21

.18

36

1.1

83

1.1

36

61

.21

18

1.1

33

81

.15

96

1.1

17

61

.12

72

1.2

95

81

.16

42

1.1

85

4

Scality

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.4

03

41

.07

32

0.9

07

60

.85

64

0.9

91

40

.86

94

0.7

67

80

.74

44

0.7

21

0.7

50

40

.82

72

0.7

16

60

.72

70

.79

80

.76

22

Scality

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

No O

bje

ct t

o e

rase

beca

use

Uplo

ad f

aile

d

[s]

Object Size [kB]


4.4

30

63

.89

24 5.0

57

84

.69

94

.59

58

4.1

98

84

.00

24

.53

02

4.1

48

4.8

50

24

.51

4.4

51

26

.50

87

.08

02

Swift

0

2

4

6

8

10

0.5 1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

[s]

Object Size [kB]


1.4

04

81

.38

38

1.3

73

81

.36

22

1.3

70

22

.07

38

1.3

89

1.3

52

63

.12

12

2.1

41

3.7

16

22

.28

32

.25

42

1.9

32 3

.18

44

Swift

Figure 6: Time to erase ten objects sequentially and in parallel into the tested storage services

13


0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Cum

ulu

s

Fake

S3

Min

io

Ria

k C

S

S3

nin

ja

S3

rver

Sca

lity

Sw

ift

[s]

Required Time to erase a Bucket (50 Tests)

Figure 7: Time to erase the bucket

ACKNOWLEDGEMENTS

This work is funded by the Hessian Ministry ofHigher Education, Research, and the Arts (’HessischesMinisterium fur Wissenschaft und Kunst’) in theframework of research for practice (’Forschung fur diePraxis’).

Many thanks to Katrin Baun for her assistance inimproving the quality of this paper.

REFERENCES

[1] B. Apperson, “Building a ceph cluster onraspberry pi,” http://bryanapperson.com/blog/the-definitive-guide-ceph-cluster-on-raspberry-pi/,accessed 12th August 2017.

[2] C. Baun, “Mobile clusters of single boardcomputers: an option for providing resources tostudent projects and researchers,” SpringerPlus,vol. 5, no. 1, 2016.

[3] C. Baun, “Performance and energy-efficiencyaspects of clusters of single board computers,”International Journal of Distributed and ParallelSystems, vol. 7, no. 2/3/4, 2016.

[4] C. Baun, M. Kunze, D. Schwab, and T. Kurze,“Octopus-a redundant array of independentservices (rais).” in CLOSER 2013: Proceedingsof the 3rd International Conference on CloudComputing and Services Science. SCITEPRESS,2013, pp. 321–328.

[5] C. Baun and R.-M. Spanou, “s3-perf source code,”https://github.com/christianbaun/s3perf, accessed12th August 2017.

[6] Z. Bjornson, “Aws s3 vs google cloudvs azure: Cloud storage performance,”http://blog.zachbjornson.com/2015/12/29/cloud-storage-performance.html, accessed 12thAugust 2017.

[7] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli,“Fog computing and its role in the internet ofthings,” in Proceedings of the first edition ofthe MCC workshop on Mobile cloud computing.ACM, 2012, pp. 13–16.

[8] Boto, “Boto source code and documentation,”https://github.com/boto/boto, accessed 12thAugust 2017.

[9] J. Bresnahan, K. Keahey, D. LaBissoniere, andT. Freeman, “Cumulus: An Open Source StorageCloud for Science,” in Proceedings of the2nd international workshop on Scientific cloudcomputing. ACM, 2011, pp. 25–32.

[10] N. Canceill, W. de Jong, and J. Saathof, “A study ofscalable storage systems,” SURF, Tech. Rep., 2014.

[11] Ceph, “Ceph Object Gateway,” http://docs.ceph.com/docs/master/radosgw/, accessed 12th August2017.

[12] COSBench, “Cloud object storage benchmark –source code and documentation,” https://github.com/intel-cloud/cosbench, accessed 12th August2017.

[13] J. Coyle, “Create a 3 node ceph storagecluster,” https://www.jamescoyle.net/how-to/1244-create-a-3-node-ceph-storage-cluster,accessed 12th August 2017.

[14] J. Coyle, “Small scale ceph replicatedstorage,” https://www.jamescoyle.net/how-to/2105-small-scale-ceph-replicated-storage,accessed 12th August 2017.

[15] G. DeCandia, D. Hastorun, M. Jampani,G. Kakulapati, A. Lakshman, A. Pilchin,

14

http://bryanapperson.com/blog/the-definitive-guide-ceph-cluster-on-raspberry-pi/

http://bryanapperson.com/blog/the-definitive-guide-ceph-cluster-on-raspberry-pi/

https://github.com/christianbaun/s3perf

http://blog.zachbjornson.com/2015/12/29/cloud-storage-performance.html

http://blog.zachbjornson.com/2015/12/29/cloud-storage-performance.html

https://github.com/boto/boto

http://docs.ceph.com/docs/master/radosgw/

http://docs.ceph.com/docs/master/radosgw/

https://github.com/intel-cloud/cosbench

https://github.com/intel-cloud/cosbench

https://www.jamescoyle.net/how-to/1244-create-a-3-node-ceph-storage-cluster

https://www.jamescoyle.net/how-to/1244-create-a-3-node-ceph-storage-cluster

https://www.jamescoyle.net/how-to/2105-small-scale-ceph-replicated-storage

https://www.jamescoyle.net/how-to/2105-small-scale-ceph-replicated-storage


S. Sivasubramanian, P. Vosshall, and W. Vogels,“Dynamo: amazon’s highly available key-valuestore,” ACM SIGOPS operating systems review,vol. 41, no. 6, pp. 205–220, 2007.

[16] Eucalyptus, “Eucalyptus source code anddocumentation,” https://github.com/eucalyptus/eucalyptus, accessed 12th August 2017.

[17] Fake S3, “Fake S3 source code anddocumentation,” https://github.com/jubos/fake-s3,accessed 12th August 2017.

[18] S. Garfinkel, “An evaluation of amazon’s gridcomputing services: Ec2, s3, and sqs,” HarvardComputer Science Group Technical Report TR-08-07, 2007.

[19] D. Guinard and V. Trifa, Building the web ofthings: with examples in node. js and raspberry pi.Manning Publications Co., 2016.

[20] HPL, “A Portable Implementation of theHigh-Performance Linpack Benchmarkfor Distributed-Memory Computers,”http://www.netlib.org/benchmark/hpl/, accessed12th August 2017.

[21] Iperf, “Iperf source code and documentation,” http://www.nwlab.net/art/iperf/, accessed 12th August2017.

[22] P. Jain, A. Goel, and S. Gupta, “Monitoring ofriak cs storage infrastructure,” Procedia ComputerScience, vol. 54, pp. 137–146, 2015.

[23] D. Karger, E. Lehman, T. Leighton, R. Panigrahy,M. Levine, and D. Lewin, “Consistent hashingand random trees: Distributed caching protocolsfor relieving hot spots on the world wide web,”in Proceedings of the twenty-ninth annual ACMsymposium on Theory of computing. ACM, 1997,pp. 654–663.

[24] K. Keahey, M. Tsugawa, A. Matsunaga, andJ. Fortes, “Sky computing,” Internet Computing,IEEE, vol. 13, no. 5, pp. 43–51, Sept 2009.

[25] Y. N. Krishnan, C. N. Bhagwat, and A. P. Utpat,“Fog computingnetwork based cloud computing,”in Electronics and Communication Systems(ICECS), 2015 2nd International Conference on.IEEE, 2015, pp. 250–251.

[26] L. Land, “Real-world benchmarkingof cloud storage providers: Amazons3, google cloud storage, and azureblob storage,” https://lg.io/2015/10/25/real-world-benchmarking-of-s3-azure-google-\cloud-storage.html, accessed 12th August 2017.

[27] R. McFarland, “s3-perf source code,” https://github.com/ross/s3-perf, accessed 12th August2017.

[28] Minio, “Minio source code and documentation,”https://github.com/minio/minio, accessed 12thAugust 2017.

[29] A. Nayak, A. Poriya, and D. Poojary, “Typeof nosql databases and its comparison withrelational databases,” International Journal ofApplied Information Systems, vol. 5, no. 4, pp. 16–19, 2013.

[30] Nimbus, “Nimbus source code anddocumentation,” https://github.com/nimbusproject/nimbus, accessed 12th August 2017.

[31] OpenStack Storage (Swift), “Swift source codeand documentation,” https://github.com/openstack/swift, accessed 12th August 2017.

[32] M. R. Palankar, A. Iamnitchi, M. Ripeanu, andS. Garfinkel, “Amazon s3 for science grids: a viablesolution?” in Proceedings of the 2008 internationalworkshop on Data-aware distributed computing.ACM, 2008, pp. 55–64.

[33] Python client for the Swift API, “Source codeand documentation,” https://github.com/openstack/python-swiftclient, accessed 12th August 2017.

[34] Riak CS, “Riak CS source code anddocumentation,” https://github.com/basho/riak cs,accessed 12th August 2017.

[35] s3cmd, “s3cmd source code and documentation,”https://github.com/s3tools/s3cmd, accessed 12thAugust 2017.

[36] S3ninja, “S3ninja source code and documentation,”https://github.com/scireum/s3ninja, accessed 12thAugust 2017.

[37] S3rver, “S3rver source code and documentation,”https://github.com/jamhall/s3rver, accessed 12thAugust 2017.

[38] Scality S3 Server, “S3 Server source codeand documentation,” https://github.com/scality/S3,accessed 12th August 2017.

[39] K. Skala, D. Davidovic, E. Afgan, I. Sovic,and Z. Sojat, “Scalable distributed computinghierarchy: Cloud, fog and dew computing,”Open Journal of Cloud Computing (OJCC),vol. 2, no. 1, pp. 16–24, 2015. [Online].Available: http://nbn-resolving.de/urn:nbn:de:101:1-201705194519

[40] Q. O. Snell, A. R. Mikler, and J. L. Gustafson,“Netpipe: A network protocol independentperformance evaluator,” in IASTED International

15

https://github.com/eucalyptus/eucalyptus

https://github.com/eucalyptus/eucalyptus

https://github.com/jubos/fake-s3

http://www.netlib.org/benchmark/hpl/

http://www.nwlab.net/art/iperf/

http://www.nwlab.net/art/iperf/

https://lg.io/2015/10/25/real-world-benchmarking-of-s3-azure-google-\ cloud-storage.html



https://github.com/ross/s3-perf

https://github.com/ross/s3-perf

https://github.com/minio/minio

https://github.com/nimbusproject/nimbus

https://github.com/nimbusproject/nimbus

https://github.com/openstack/swift

https://github.com/openstack/swift

https://github.com/openstack/python-swiftclient

https://github.com/openstack/python-swiftclient

https://github.com/basho/riak_cs

https://github.com/s3tools/s3cmd

https://github.com/scireum/s3ninja

https://github.com/jamhall/s3rver

https://github.com/scality/S3

http://nbn-resolving.de/urn:nbn:de:101:1-201705194519



Conference on Intelligent InformationManagement and Systems, vol. 6. USA,1996.

[41] Stanchion, “Stanchion source code anddocumentation,” https://github.com/basho/stanchion, accessed 12th August 2017.

[42] Swift3, “Swift3 middleware for openstack swift -source code and documentation.”

[43] N. Tiwari, “Enterprise-grade cloudstorage with nginx plus andminio,” https://www.nginx.com/blog/enterprise-grade-cloud-storage-nginx-plus-minio/,accessed 24th June 2017.

[44] R. Vijayan, “Ceph on a raspberrypi,” https://www.linkedin.com/pulse/ceph-raspberry-pi-rahul-vijayan, accessed 12thAugust 2017.

[45] Y. Wang, “Definition and categorization of dewcomputing,” Open Journal of Cloud Computing(OJCC), vol. 3, no. 1, pp. 1–7, 2016. [Online].Available: http://nbn-resolving.de/urn:nbn:de:101:1-201705194546

[46] S. A. Weil, S. A. Brandt, E. L. Miller, D. D.Long, and C. Maltzahn, “Ceph: A scalable,high-performance distributed file system,” inProceedings of the 7th symposium on Operatingsystems design and implementation. USENIXAssociation, 2006, pp. 307–320.

[47] C. W. Zhao, J. Jegatheesan, and S. C. Loon,“Exploring iot application using raspberry pi,”International Journal of Computer Networks andApplications, vol. 2, no. 1, pp. 27–34, 2015.

[48] Y. Zhao, S. Li, S. Hu, H. Wang, S. Yao,H. Shao, and T. Abdelzaher, “An experimentalevaluation of datacenter workloads on low-powerembedded micro servers,” Proceedings of theVLDB Endowment, vol. 9, no. 9, pp. 696–707,2016.

[49] Q. Zheng, H. Chen, Y. Wang, J. Duan, andZ. Huang, “Cosbench: A benchmark tool for cloudobject storage services,” in Cloud Computing(CLOUD), 2012 IEEE 5th InternationalConference on. IEEE, 2012, pp. 998–999.

[50] Q. Zheng, H. Chen, Y. Wang, J. Zhang, andJ. Duan, “Cosbench: cloud object storage

benchmark,” in Proceedings of the 4th ACM/SPECInternational Conference on PerformanceEngineering. ACM, 2013, pp. 199–210.

AUTHOR BIOGRAPHIES

Dr. Christian Baun is aProfessor at the Facultyof Computer Science andEngineering of the FrankfurtUniversity of Applied Sciencesin Frankfurt am Main, Germany.He earned his Diploma degree inInformatik (Computer Science)in 2005 and his Master degreein 2006 from the MannheimUniversity of Applied Sciences.

In 2011, he earned his Doctor degree from the Universityof Hamburg. He is author of several books, articles andresearch papers. His research interest includes operatingsystems, distributed systems and computer networks.

Henry-Norbert Cocos studiescomputer science at theFrankfurt University of AppliedSciences. His research interestincludes distributed systemsand single board computers.Currently, he constructs a 256node cluster of Raspberry Pi 3nodes which shall be usedto analyze different parallelcomputation tasks. For thiswork, he analyzes which

administration tasks need to be carried out during thedeployment and operation phase and how these taskscan be automated.

Rosa-Maria Spanou studiescomputer science at theFrankfurt University of AppliedSciences. Her research interestincludes distributed systemsand single board computers.Currently, she constructs andanalyzes different multi-nodeobject-based cloud storagesolutions.

16

https://github.com/basho/stanchion

https://github.com/basho/stanchion

https://www.nginx.com/blog/enterprise-grade-cloud-storage-nginx-plus-minio/

https://www.nginx.com/blog/enterprise-grade-cloud-storage-nginx-plus-minio/

https://www.linkedin.com/pulse/ceph-raspberry-pi-rahul-vijayan

https://www.linkedin.com/pulse/ceph-raspberry-pi-rahul-vijayan



performance aspects of object-based storage services on ... · a network attached storage (nas) or...

Documents