performance analysis: benchmarks of bare-metal & virtual ... · performance testing and...

Performance Analysis: Benchmarks of Bare-Metal & Virtual Clouds Benchmarking offerings from Internap, Amazon Web Services and Rackspace By Cloud Spectator October 2013

Cloud Spectator Performance Report: Internap, Amazon and Rackspace October 2013

1

Table of Contents

Abstract and Introduction 2 Executive Summary 4 System Performance 5 CPU Performance 6 RAM Performance 9 Internal Network Performance 10 Disk Performance 12 Database Performance 17

Price-Performance 18 Conclusion 20 Appendices 21 About Cloud Spectator 30


2

Abstract Virtualized infrastructure-as-a-service (IaaS) has drawn increasing attention and usage due to its granular billing, ease-of-use, and broad network access. Bare-metal cloud services, which are essentially physical servers that can be deployed on demand and billed hourly, can offer significant improvements over virtualized IaaS in performance, consistency and cost efficiency for many applications. Benchmark tests comparing similarly sized virtual and bare-metal cloud configurations reveal that bare-metal cloud yields superior CPU, RAM, storage, and internal network performance. Predictably, solid-state drive (SSD) configurations showed substantially higher disk performance measures when compared with spinning serial ATA (SATA) drives, although processor, memory and network results between the two configurations were virtually identical. Cost efficiency, measured by performance-weighting the prices, was also measurably higher with the tested bare-metal cloud deployment inferring superior value from bare-metal cloud, all-else-equal. Introduction Bare-metal Cloud Bare-metal cloud offerings refer to a type of high-performance hosted-server offering that combines the advantages of traditional dedicated servers with those of IaaS virtual servers. With bare-metal cloud, the entire physical server’s resources are dedicated to a single user, purportedly offering better performance than a comparable virtualized server. Since resources are not being shared, no hypervisor layer is needed, thereby allowing more of the server’s processing power to be devoted to the application. Unlike typical dedicated servers, which can take days or weeks to deploy, bare-metal cloud offerings can be provisioned in a matter of minutes. Bare-metal cloud servers are also able to be implemented using Application Programming Interface (API) methods or management portals and billed on a usage-based model. Similar to cloud IaaS virtual offerings, bare-metal cloud is billed only when the server is in use. This flexibility allows bare metal to accommodate for peak demand quickly similar to virtual cloud servers. Afterwards, additional resources can be deleted, with the customer only being charged for the hours of additional resources, compared to monthly fees typical with traditional dedicated servers. This study tests the degree of performance improvement and value enhancement that can be obtained using bare-metal cloud relative to two well-known virtualized IaaS offers.

Bare metal vs. Virtual The difference between bare-metal cloud servers and virtual servers comes down to two key elements: the hypervisor and multi-tenancy. The hypervisor is used to virtualize the resources of physical machines, creating multiple virtual machines on each physical server for a multi-tenant environment. Bare-metal servers do not have a hypervisor; instead, the entire physical server’s resources are reserved for a single tenant. However, bare-metal cloud is similar to virtual cloud in that the physical servers are linked together with an orchestration layer, so that the resources can be provisioned and scaled quickly using an API or other remote management tool. The level of flexibility for both offerings is similar as both operate in a usage-based billing model and offer similar features and services. It is commonly known that adding a hypervisor layer will result in some performance degradation for the virtual servers. Also, multi-tenancy and oversubscription of some services can cause an individual user to fight for shared resources on the server. In contrast, bare-metal cloud resources are fully dedicated to a single user. It follows that bare-metal cloud users routinely report greater processing power with more consistent disk and network I/O performance.


3

The Study Performance Testing Cloud Spectator monitors the CPU, RAM, storage, and internal network performance of over 20 of the world’s most well-known IaaS services to understand important aspects of virtual server performance. Tests are run at least three times per day, 365 days per year to capture variability in addition to performance level. Tests are chosen based on reliability and practicality. The goal is to provide an indication of where certain providers perform well relative to others. This can give consumers an indication of which services would be best for their application(s) by understanding the performance of provider resources most critical to that application. Singular benchmarks alone should not be the only deciding factor in the provider selection process. A feature set, configuration matches, pricing and ancillary services such as security, compliance, and disaster recovery should always factor into any vendor selection process. However, performance is a very important piece to the puzzle.

The Comparison For the purpose of generating this document, Cloud Spectator measured the performance of Internap’s bare-metal cloud offering against virtual offerings from Amazon and Rackspace. The goal was to quantify how much of a performance penalty users are taking by choosing virtual cloud servers. Over a period of ten days, Cloud Spectator ran benchmark tests across Internap bare-metal cloud, AWS Elastic Compute Cloud (EC2) with Amazon Elastic Block Storage (EBS), and Rackspace OpenCloud with Rackspace Block Storage. Each test was run to understand the unique performance capabilities of each offering’s CPU, internal network, RAM and disk. Cloud Spectator accounted for performance capability and stability for each provider to understand the value each one delivers to its users. Tests were run on 8GB servers for Internap and Rackspace, and 7.5GB servers for Amazon (for more details, see Methodology section in Appendix B). While other factors can sway the perception of provider comparisons, including value-added services, features, support, security, etc., Cloud Spectator chose to focus objectively on performance to derive numerical relationships between the levels of service delivered from each provider. Performance testing and benchmarking of cloud computing platforms is a complex task, compounded by the differences between providers and the use cases of cloud infrastructure users. IaaS services are utilized by a large variety of industries, and performance metrics cannot be completely understood by simply representing cloud performance with a single value. When selecting a cloud computing provider, IT professionals consider many factors: feature-sets, cost, security, location and more. However, performance is a key issue that drives many others, including cost. In many cases, four primary resources affect overall server performance: central processing unit (CPU), random access memory (RAM), disk, and internal network. These four resources are what this comparison focuses on.

SSD vs. SATA A growing number of service providers are now offering their customers the option to choose between solid-state drives (SSD) or traditional SATA hard drives. While the focus of this study centers on comparing similarly priced virtual instances to Internap bare-metal cloud servers outfitted with a SSD, it also assesses the relative performance of a lower-priced, bare-metal SATA configuration. The results display that while CPU, RAM, and internal network performance are comparable between the bare-metal SATA and SSD instances, the disk tests show significant variation. Certain CPU or RAM-intense scenarios that require less disk IO could benefit from the lower-cost SATA option relative to SSD and/or virtual cloud instances (see Appendix A).


4

Executive Summary Overall Performance Internap bare-metal cloud has both higher performance and stability than Amazon and Rackspace in nearly 70% of the benchmark tests across the 10-day testing period. System Performance Internap bare-metal cloud has both the highest performance and stability in the general system test.

Internap bare-metal cloud scored 7x higher than the second best performing provider. Internap bare-metal cloud and Rackspace have very stable system performance: Coefficient of Variation (CV)

less than 1%. Internap bare-metal cloud outperformed Rackspace by 7x and Amazon by 10.1x in the system test.

CPU Performance Internap bare-metal cloud has both the highest performance and stability across all the CPU tests.

Internap bare-metal cloud outperformed Amazon and Rackspace in 3 out of 3 CPU tests. Internap bare-metal cloud was more stable than Amazon and Rackspace in 3 out of 3 CPU tests. Internap bare-metal cloud outperformed Rackspace by an average 2.5x and Amazon by 4.4x in the CPU tests.

RAM Performance Internap bare-metal cloud has the highest performance across the RAM test and is also the most stable.

Internap bare-metal cloud outperformed both Amazon and Rackspace in the RAM test. Internap bare-metal cloud has very stable RAM performance: CV of 1% or less. Internap bare-metal cloud outperformed Rackspace by 1.7x and Amazon by 2.1x in the RAM test.

Internal Network Performance Internap bare-metal cloud has the highest performance and stability across the internal network tests, showing high throughput and low latency.

Internap bare-metal cloud outperformed Amazon and Rackspace in 2 out of 2 internal network tests. Internap bare-metal cloud was more stable than Amazon and Rackspace in 2 out of 2 internal network tests. Internap bare-metal cloud outperformed Rackspace by an average 2.5x and Amazon by 2.5x in the internal

network tests. Disk Performance Internap bare-metal cloud has both the highest performance and stability in 60% of the disk performance tests.

Internap bare-metal cloud outperformed Amazon and Rackspace in 4 out of 5 disk tests. Internap bare-metal cloud was more stable than Amazon and Rackspace in 4 out of 5 disk tests. Internap bare-metal cloud outperformed Rackspace by an average 8.4x and Amazon by 4.9x in the disk tests.

Database Performance Internap bare-metal cloud has the highest performance across the database test and has the second highest stability.

Internap bare-metal cloud outperformed Amazon and Rackspace in the database test. Internap bare-metal cloud was more stable than Amazon in the database test. Internap bare-metal cloud outperformed Rackspace by 67% and Amazon by 65% in the database test.


5

System Performance UnixBench Test Description Unixbench runs a set of individual benchmark tests, aggregates the scores, and creates a final score to gauge the performance of Linux systems. According to the Unixbench homepage (https://code.google.com/p/byte-unixbench/):

“The purpose of UnixBench is to provide a basic indicator of the performance of a Unix-like system; hence, multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value. The entire set of index values is then combined to make an overall index for the system. Some very simple graphics tests are included to measure the 2D and 3D graphics performance of the system. Multi-CPU systems are handled. If your system has multiple CPUs, the default behavior is to run the selected tests twice - once with one copy of each test program running at a time, and once with N copies, where N is the number of CPUs. This is designed to allow you to assess: - the performance of your system when running a single task - the performance of your system when running multiple tasks - the gain from your system's implementation of parallel processing Do be aware that this is a system benchmark, not a CPU, RAM or disk benchmark. The results will depend not only on your hardware, but on your operating system, libraries, and even compiler.”

System Performance Results The results of the Unixbench benchmark test show Internap bare-metal cloud outperforming Amazon and Rackspace in speed and stability. Internap’s bare-metal cloud has an average score of 6415, outperforming Rackspace’s average of 917 by 7x, and Amazon’s average of 637 by 10.1x. For performance stability, Internap bare-metal cloud has the least variable performance, with the lowest Coefficient of Variation (CV) of 0.4%, followed by Rackspace with 0.8%, and Amazon with 9.9%.

0

1000

2000

3000

4000

5000

6000

7000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Sco

re

SYSTEM TEST - Unixbench:Internap bare-metal vs. Amazon vs. Rackspace

Amazon

Internap Bare Metal

Rackspace

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 6415 24 0.4% 6474 6358 Amazon 637 63 9.9% 784 554 Rackspace 917 7 0.8% 931 902

The graph above compares system performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the UnixBench benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the test period.


6

CPU Performance File Compression Test Description With p7zip’s integrated benchmark feature, we test the performance of the virtual CPU by measuring the millions of instructions per second (MIPS) that it can handle when compressing a file. This benchmark test is a multi-threaded CPU test.

File Compression Performance Results The results of the File Compression benchmark show Internap bare-metal cloud outperforming Amazon and Rackspace in speed and stability. Internap bare-metal cloud has an average speed of 17626 MIPS, outperforming Rackspace’s average of 5983 MIPS by 3x and Amazon’s average of 2831 MIPS by 6.2x. For performance stability, Internap bare-metal cloud has the lowest Coefficient of Variation (CV) of 1.8%, followed by Rackspace with 2.9%, and Amazon with 14.3%.

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

MIP

S

CPU TEST - File Compression:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 17626 MIPS 322 MIPS 1.8% 18202 MIPS 16976 MIPS Amazon 2831 MIPS 405 MIPS 14.3% 3505 MIPS 2136 MIPS Rackspace 5983 MIPS 173 MIPS 2.9% 6254 MIPS 5610 MIPS

The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the File Compression benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the test period.


7

Audio Encoding Test Description From the Phoronix Benchmark Test Suite, this test times how long it takes to encode a WAV file to MP3 format using LAME, an MP3 encoder. This encoding test is single-threaded, indicating the CPU performance of a single core on each provider. To compare the performance difference between single-core and multi-core encoding of each provider, see results for CPU Test: Video Encoding. Audio Encoding Performance Results The results of the Audio Encoding benchmark show Internap bare-metal cloud outperforming Amazon and Rackspace in speed and stability. Internap bare-metal cloud has an average time of 15 seconds, outperforming Rackspace’s average of 34 seconds, and Amazon’s average of 37 seconds. For performance stability, Internap bare-metal cloud has the lowest CV of 0.1%, followed by Rackspace with 1.6% and Amazon with 9.9%.

0

5

10

15

20

25

30

35

40

45

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Sec

on

ds

CPU TEST - Audio Encoding:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 15 Seconds 0 Seconds 0.1% 15 Seconds 15 Seconds Amazon 37 Seconds 4 Seconds 9.9% 40 Seconds 30 Seconds Rackspace 34 Seconds 1 Seconds 1.6% 36 Seconds 34 Seconds

The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the Audio Encoding benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the test period.


8

Video Encoding Test Description By using the x264 Video Encoding Test within the Phoronix Benchmark Suite, we can test the performance of the CPU through the compute-intensive task of converting video, and score the provider’s virtual CPU performance based on how many frames are converted per second, i.e., Frames Per Second (FPS). To compare the performance difference between single-core and multi-core encoding of each provider, see results for CPU Test: Audio Encoding. Video input format: YUV4MPEG2 Video output format: H.264/MPEG-4 AVC format Video Encoding Performance Results The results of the Video Encoding benchmark show Internap bare-metal cloud outperforming Amazon and Rackspace in speed and stability. Internap bare-metal cloud has an average of 132 FPS, outperforming Rackspace’s average of 44 FPS by 3x and Amazon’s average of 24 FPS by 5.5x. For performance stability, Internap bare-metal cloud has the lowest CV of 0.5%, followed by Rackspace with 3.8%, and Amazon with 9.0%.

0

20

40

60

80

100

120

140

160

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Fra

mes

Per

Sec

on

d

CPU TEST - Video Encoding:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 132 FPS 1 FPS 0.5% 134 FPS 131 FPS Amazon 24 FPS 2 FPS 9.0% 29 FPS 21 FPS Rackspace 44 FPS 2 FPS 3.8% 46 FPS 41 FPS

The graph above compares CPU performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the Video Encoding benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the test period.


9

RAM Performance RAMSpeed SMP Test Description The RAMSpeed test is an aggregate of several tests that measure COPY, SCALE, ADD and TRIAD functions for both integer and floating point values. More information on COPY, SCALE, ADD, and TRIAD:

COPY transfers data from one memory location to another (A = B)

SCALE multiplies the data with a constant value before writing it (A = Bn)

ADD reads results from two different locations, adds those results and writes then to the new location (A = B + C)

TRIAD merges ADD and SCALE. It reads data from the first memory location, scales it (multiplies it), then adds data from the second one and writes to the new location (A = Bn + C)

Each test results in a score in MB/s. All scores are averaged to come up with the final score. RAMspeed SMP Performance Results The results of the RAMspeed Test show Internap bare-metal cloud with the highest performance and stability throughout the testing period, followed by Rackspace, then Amazon. Internap bare-metal cloud has an average of 12544 Mb/s, outperforming Rackspace’s average of 7299 Mb/s by 72% and Amazon’s average of 6007 Mb/s by 2.1x. Internap bare-metal cloud has nearly zero instability with a CV of 0.1%, followed by Rackspace with a CV of 2.4% and Amazon proving its instability with a CV of 45.2%.

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

MiB

ytes

/sec

RAM TEST - RAMspeed SMP Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 12544 Mb/sec 16 Mb/sec 0.1% 12557 Mb/sec 12507 Mb/sec Amazon 6007 Mb/sec 2716 Mb/sec 45.2% 15476 Mb/sec 3032 Mb/sec Rackspace 7299 Mb/sec 175 Mb/sec 2.4% 7526 Mb/sec 6894 Mb/sec

The graph above compares RAM performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the RAMspeed SMP benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the majority of the test period.


10

Internal Network Performance Iperf Test Description Running Iperf tests the network throughput between two virtual machines (VMs) within the same private network inside the data center. The results are important in understanding the possible network bottlenecks. Applications that require databases, which may pull from storage located off of the local server, need a large pipe to transfer data as efficiently as possible to load quickly from the server side. This is especially true for big data applications like Hadoop, which push massive amounts of data. Throughput and bandwidth are often confused; while both measure the size of the pipe, bandwidth is the theoretical capacity, while throughput is the actual bandwidth the user actually receives. This is an important distinction for a cloud environment, where a 10GBit pipe may be split among many users, thereby resulting in lower actual throughput per user.

Internal Network Throughput Results The results of the Iperf benchmark show Internap bare-metal cloud with the highest throughput speed and stability. Internap bare-metal cloud has an average network throughput of 913 Mbits/sec, outperforming Amazon’s average of 289 Mbits/sec by 3.2x and Rackspace’s average of 288 Mbits/sec by 3.2x. Internap bare-metal cloud has zero instability with a CV of 0.0%. Amazon and Rackspace are both stable, with CVs of 1.5% and 0.6% respectively.

0

100

200

300

400

500

600

700

800

900

1000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Mib

its/

sec

NETWORK TEST - Iperf Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 913 Mbits/sec 0 Mbits/sec 0.0% 913 Mbits/sec 913 Mbits/sec Amazon 289 Mbits/sec 4 Mbits/sec 1.5% 294 Mbits/sec 276 Mbits/sec Rackspace 288 Mbits/sec 2 Mbits/sec 0.6% 291 Mbits/sec 283 Mbits/sec

The graph above compares internal network performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the Iperf benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the test period. Amazon and Rackspace’s performance are nearly identical, resulting in Rackspace’s trendline to overlap Amazon’s in the graph above.


11

Ping Test Description In contrast to throughput testing, ping commands can be used to measure the latency between nodes on a private network. Higher latency severely degrades application performance in a distributed computing environment, and can adversely impact user experience, especially with regard to time-sensitive applications. 1000 microseconds is 1 millisecond. Cloud Spectator runs the ping test from the Linux Terminal with the following parameters: ping –c 100 –i 0.25 –s 0 –w 300 –W 5 Latency Results The results of the Ping benchmark show the relative stability of Internap Bare Metal, while Amazon and Rackspace’s performance was erratic and unstable for the duration. Internap bare-metal cloud has the lowest average latency of 271 microseconds, performing 80% better than Rackspace’s average latency of 1381 microseconds, and 87% better than Amazon’s average latency of 2153 microseconds. Internap bare-metal cloud has the most stable latency with a CV of 0.8%, followed Amazon with 22.2% and Rackspace with 36.4%.

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 271 µs 2 µs 0.8% 274 µs 265 µs Amazon 2153 µs 478 µs 22.2% 3823 µs 1825 µs Rackspace 1381 µs 502 µs 36.4% 2760 µs 699 µs

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Mic

rose

con

ds

NETWORK TEST - Ping:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

The graph above compares internal network performance on a Linux server between Internap bare-metal, Amazon and Rackspace. The graph shows the scores from the Ping benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal has the least amount of latency compared to Amazon and Rackspace over the test period.


12

Disk Performance

IOzone Test Description IOzone measures HDD Performance read and write speed. This test produces measurements from a variety of file operations and is used in the general file system analysis of a system. The average between HDD read and write speeds make up the final score. IOzone performs the following operations: read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread, nmap, aio_read, and aio_write. IOzone Performance Results The results of the IOzone benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, outperforming Amazon and Rackspace by about 6x. Internap bare-metal cloud leads with an average of 4453 Mb/s, outperforming Amazon’s average of 703 Mb/s by 6.3x, and Rackspace’s average of 700 Mb/s by 6.4x. Internap bare-metal cloud has the most stable performance for the testing period with a CV of 1.7%, followed by Rackspace with 6.9% and Amazon with 13.0%.

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 4453 Mb/sec 74 Mb/sec 1.7% 4536 Mb/sec 4279 Mb/sec Amazon 703 Mb/sec 91 Mb/sec 13.0% 938 Mb/sec 619 Mb/sec Rackspace 700 Mb/sec 48 Mb/sec 6.9% 726 Mb/sec 531 Mb/sec

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

MiB

ytes

/sec

DISK TEST - IOzone Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

The graph above compares disk performance on a Linux server between Internap bare-metal, Amazon and Rackspace. The graph shows the scores from the IOzone benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal outperforms Amazon and Rackspace over the test period.


13

MongoPerf – Read w/out Cache Test Description Cloud Spectator tests disk read operations per second with Mongoperf, MongoDB’s native benchmarking tool for gauging disk IO performance. By eliminating the cache functionality, data must always be read directly from the disk, eliminating the added benefit of RAM, for a true measure of disk IO. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nThreads:32, fileSizeMB:2000, mmf:false, r:true, w:false, recSize:4, syncDelay:1 MongoPerf – Read w/out Cache Performance Results

The results of the Read without Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by Amazon then Rackspace. Internap bare-metal cloud has the highest average of 20045 Ops/sec, outperforming Amazon’s average of 5399 Ops/sec by 3.7x, and Rackspace’s average of 693 Ops/sec by 28.9x. While having the lowest performance, Rackspace has the most stable performance throughout the testing period with a CV of 17.6%, followed by Internap bare-metal cloud with a CV of 22.3%, and Amazon proving to be very unstable with a CV of 75.6%.

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 20045 Ops/sec 4469 Ops/sec 22.3% 27003 Ops/sec 10406 Ops/sec Amazon 5399 Ops/sec 4084 Ops/sec 75.6% 16663 Ops/sec 507 Ops/sec Rackspace 693 Ops/sec 122 Ops/sec 17.6% 1136 Ops/sec 563 Ops/sec

0

5000

10000

15000

20000

25000

30000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Op

erat

ion

s/se

c

DISK TEST - Read w/out Cache Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

The graph above compares disk performance on a Linux server between Internap bare-metal, Amazon and Rackspace. The graph shows the scores from the Read w/out Cache benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal outperforms Amazon and Rackspace over the test period.


14

MongoPerf – Write w/out Cache Test Description

Cloud Spectator tests disk write operations per second with Mongoperf, MongoDB’s native benchmarking tool for gauging disk IO performance. By eliminating the cache functionality, data must always be written directly to the disk, eliminating the added benefit of RAM, for a true measure of disk IO. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nThreads:32, fileSizeMB:2000, mmf:false, r:false, w:true, recSize:4, syncDelay:1 MongoPerf – Write w/out Cache Performance Results

The results of the Write without Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by Rackspace, and then Amazon. Internap bare-metal cloud has an average of 2292 Ops/sec, outperforming Rackspace’s average of 1500 Ops/sec by 53%, and Amazon’s average of 1157 Ops/sec by 98%. Internap bare-metal cloud also has the most stable performance throughout the testing period with a CV of 0.5%, followed by Rackspace with 7.1%, and Amazon with 9.5%.


0

500

1000

1500

2000

2500

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Op

erat

ion

s/se

c

DISK TEST - Write w/out Cache Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the Write without Cache benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the test period.


15

MongoPerf – Read w/ Cache Test Description

Cloud Spectator tests disk read operations per second with Mongoperf, MongoDB’s native benchmarking tool for gauging disk IO performance. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nThreads:32, fileSizeMB:2000, mmf:true, r:true, w:false, recSize:4, syncDelay:60 MongoPerf – Read w/ Cache Performance Results

The results of the Read with Cache benchmark show Amazon with the highest performance throughout the testing period, followed by Internap Bare Metal, and then Rackspace. Amazon has an average of 1629198 Ops/sec, outperforming Internap Bare Metal’s average of 1095523 Ops/sec by 49%, and Rackspace’s average of 274943 Ops/sec by 5.9x. While Amazon is the highest performing, it is also the least stable with a CV of 31.9%. Internap bare-metal cloud is the most stable with a CV of 1.3%, followed by Rackspace with 8.8%.

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 1095523

Ops/sec 14418 Ops/sec 1.3% 1126520 Ops/sec 1072470 Ops/sec

Amazon 1629198 Ops/sec

520328 Ops/sec 31.9% 2375360 Ops/sec 593233 Ops/sec

Rackspace 274943 Ops/sec 24261 Ops/sec 8.8% 333211 Ops/sec 222528 Ops/sec

0

500000

1000000

1500000

2000000

2500000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Op

erat

ion

s/se

c

DISK TEST - Read w/ Cache Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the Read w/ Cache benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Amazon outperforms Internap bare-metal cloud and Rackspace over the majority of the test period. Amazon is also the most variable over the test period compared to the other providers.


16

MongoPerf – Write w/ Cache Test Description

Cloud Spectator tests disk write operations per second with Mongoperf, MongoDB’s native benchmarking tool for gauging disk IO performance. Database environments are particularly concerned about disk IO because of the frequency of read and write operations. If the disk is slow, the entire application may lag while waiting. The parameters run: nThreads:32, fileSizeMB:2000, mmf:true, r:false, w:true, recSize:4, syncDelay:60 MongoPerf – Write w/ Cache Performance Results

The results of the Write with Cache benchmark show Internap bare-metal cloud with the highest performance throughout the testing period, followed by Rackspace, then Amazon. Internap bare-metal cloud has an average of 205238 Ops/sec, outperforming Rackspace’s average of 163262 Ops/sec by 26%, and Amazon’s average of 17239 Ops/sec by 11.9x. Internap bare-metal cloud has the most stable performance throughout the test period with a CV of 1.1%, followed by Amazon with 24.5% and Rackspace proving to be very unstable with a CV of 60.0%.


0

50000

100000

150000

200000

250000

300000

350000

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Op

erat

ion

s/se

c

DISK TEST - Write w/ Cache Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

The graph above compares disk performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the Write w/ Cache benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the majority of the test period.


17

Database Performance MySQL Test Description The MySQL Performance test is MySQL’s native benchmarking tool. This tests time to completion of MySQL’s database queries/commands (alter, insert, update, and delete). Lower scores are better. MySQL Performance Results The results of the MySQL Performance benchmark show Internap bare-metal cloud outperforming Amazon and Rackspace throughout the testing period. Internap bare-metal cloud outperformed the other offerings by approximately 3x with an average of 6 seconds, with Amazon and Rackspace following with average scores of 17 seconds, 18 seconds respectively. Rackspace has the most stable performance with a CV of 6.3%, followed by Internap bare-metal cloud with 9.5%, and Amazon with 12.0%.

PROVIDER AVERAGE STANDARD DEVIATION CV 10-Day HIGH 10-Day LOW Internap 6 Seconds 1 Seconds 9.5% 7 Seconds 4 Seconds Amazon 17 Seconds 2 Seconds 12.0% 20 Seconds 12 Seconds Rackspace 18 Seconds 1 Seconds 6.3% 21 Seconds 16 Seconds

0

5

10

15

20

25

Sep

26

Sep

27

Sep

28

Sep

29

Sep

30

Oct

1

Oct

2

Oct

3

Oct

4

Oct

5

Sec

on

ds

DATABASE TEST - MySQL Performance Test:Internap bare-metal cloud vs. Amazon vs. Rackspace

Amazon

Internap

Rackspace

The graph above compares database performance on a Linux server between Internap bare-metal cloud, Amazon and Rackspace. The graph shows the scores from the MySQL Performance benchmark test for each provider over a period of 10 days, with two data points shown for each day. As shown above, Internap bare-metal cloud outperforms Amazon and Rackspace over the test period.


18

Price-Performance

Measuring Value Raw performance numbers do not always tell the full story. While some users may have performance as their first priority and be willing to pay extra for a quality solution, others have value in mind. These users try to understand which services give the best resource performance for their dollar. Users focused solely on price may not get the best value. If one virtual server has twice the CPU performance of another with an identical configuration, but is priced only 50% more, then this offering provides better value to the user. Instead of paying for two or more servers to get sufficient performance, the user only pays for one and saves money. Extrapolating this idea to a much bigger distributed environment like a Hadoop cluster enhances the importance of accounting for price-performance value. The Calculation After obtaining the correct performance metrics, the calculation for each test is quite simple: Performance/price = “price-performance” value To further simplify, we take the highest price-performance metric and divide that into all other price-performance metrics for the given test. Then we multiply by 100. This gives the offering providing the most value for that test a score of 100, and pegs all other offerings in relativity to that top score of 100. We call this value out of 100 the CloudSpecs Score. This represents our final price-performance score. For example, this is how we calculate price-performance for the UnixBench test:

Offering Score Divided by Price Divided by Highest Score Multiply by 100 UnixBench CloudSpecs Score

Internap bare-metal cloud ($0.36/hour)

6,415/$0.36 = 17,818 17,818/17,818 = 1.000 1.000*100 = 100.0 100.0

Amazon ($0.25/hour) 637/$0.25 = 2,548 2,548/17,818 = 0.143 0.143*100 = 14.3 14.3

Rackspace ($0.49/hour) 917/$0.49 = 1,871 1,871/17,818 = 0.105 0.105*100 = 10.5 10.5

These scores indicate that for the UnixBench test, Internap’s bare-metal cloud provides:

7x more value than Amazon

9.5x more value than Rackspace The Results The following tables provide price-performance results for all tests involving Internap bare-metal cloud servers outfitted with SSD versus nominal virtual instances of Amazon and Rackspace.

System CPU RAM UnixBench Compression Audio Encoding Video Encoding RAMSpeed

INTERNAP 100.0 100.0 100.0 100.0 100.0

AMAZON 14.3 23.1 56.6 26.2 69.0

RACKSPACE 10.5 24.9 31.4 24.6 42.8


19

Storage

IOzone Read(w/o Cache) Write(w/o Cache) Read(w/ Cache) Write(w/ Cache)

INTERNAP 100.0 100.0 100.00 46.7 100.0

AMAZON 22.7 38.8 72.70 100.0 12.1

RACKSPACE 11.6 2.5 48.08 8.6 58.4

Internal Network Database Iperf Ping MySQL

INTERNAP 100.0 100.0 100.0

AMAZON 45.6 18.1 51.0

RACKSPACE 23.1 14.4 24.3 Results Summary Internap’s bare-metal cloud offers the best price-performance ratio in 12 out of 13 total tests.

1/1 system test 3/3 CPU tests 1/1 RAM test 4/5 storage tests 2/2 internal network tests 1/1 database test


20

Conclusion Study Results In terms of raw performance, UnixBench, the lone system test in this study, shows Internap bare-metal cloud surpassing the next best virtualized option, Rackspace Cloud, by 7 times. Although bare-metal did not perform the best in every test, it outperformed virtual servers by a significant margin in most. It can therefore be concluded from the data that users will realize much higher performance in a bare-metal cloud offering in each resource. Along with the raw performance numbers, price-performance value was also examined. Although the bare-metal cloud server was priced higher than one of the virtual IaaS providers tested (Amazon), the results show that its relative performance more than compensates for its higher price. Bare metal offers over 6 times more price-performance value than any other offering and provides more value in 12 of 13 total tests. As indicated by the results in Appendix A, lower priced bare metal instances leveraging SATA drives can also yield significantly more value than virtualized IaaS in scenarios where storage access is less intensive. Implications for IaaS Users As the data show, bare-metal cloud could provide users the opportunity to derive more value for certain IaaS workloads. With the option to use a dedicated server as they would a cloud server, users can potentially better manage workloads that typically require direct access to physical hardware, such as databases and calculation-intensive applications. In addition, organizations that have limited their IT deployments to long-term hosted or owned environments due to performance concerns can look at bare-metal cloud as a way to maintain quality while improving agility and asset efficiency. Areas for Additional Research Possible areas for future study include testing performance differentials across specific use cases to better understand where bare metal and virtualized IaaS provide the best price/value fit, expanding the value equation to include additional services such as on-demand load balancing, back-up and storage, database services etc.


21

Appendix A: Traditional SATA Versus SSD & Virtual Instances

SATA – Bare-metal Performance The bare-metal cloud instances with SATA drives perform similarly to the bare-metal cloud instances with SSD drives in comparison to the virtual instances offered by Amazon and Rackspace. Because both of the bare-metal instances have the same CPU, RAM and internal network, performance was comparable for the two instances. In the disk benchmarks, the bare-metal SATA instance performed comparably to significantly lower than the bare-metal SSD instances. The stability of performance was comparable between the bare-metal instances for the majority of tests as well. System Performance Results The results of the Unixbench benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace in general system performance. The Internap SATA instance performed higher than the Internap SSD instance by 3%, Amazon by 10.3x and Rackspace by 7.2x. Internap SATA, Internap SSD and Rackspace had CVs under 1%, while Amazon reached 9.9%.

File Compression Performance Results The results of the File Compression benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 1%, but higher than Amazon by 6.2x and Rackspace by 3x. Internap SATA, Internap SSD and Rackspace had CVs under 3%, while Amazon reached 14.3%.

0

2000

4000

6000

8000

Ave

rag

e S

core

SYSTEM TEST - UnixBench:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

5.0%

10.0%

15.0%Co

effi

cen

t o

f V

aria

tio

n

SYSTEM TEST - UnixBenchPerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE

0

5000

10000

15000

20000

Ave

rag

e M

IPS

CPU TEST - File Compression:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

5.0%

10.0%

15.0%

20.0%Co

effi

cen

t o

f V

aria

tio

n

CPU TEST - File CompressionPerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE


22

Video Encoding Performance Results The results of the Video Encoding benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 2%, but higher than Amazon by 5.4x and Rackspace by 2.9x. Internap SATA, Internap SSD had CVs less than 1%, while Rackspace was 3.8% and Amazon was 9%.

Audio Encoding Performance Results The results of the Audio Encoding benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 1%, but higher than Amazon by 60% and Rackspace by 57%. Internap SATA, Internap SSD and Rackspace had CVs under 2%, while Amazon reached 9.9%.

0

50

100

150

Ave

rag

e F

PS

CPU TEST - Video Encoding:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

5.0%

10.0%Co

effi

cen

t o

f V

aria

tio

n

CPU TEST - Video EncodingPerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE

0

10

20

30

40

Ave

rag

e S

eco

nd

s

CPU TEST - Audio Encoding:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

5.0%

10.0%

15.0%Co

effi

cen

t o

f V

aria

tio

n

CPU TEST - Audio EncodingPerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE


23

RAMSpeed SMP Performance Results The results of the RAMSpeed SMP benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 1%, but higher than Amazon by 2.1x and Rackspace by 72%. Internap SATA, Internap SSD and Rackspace had CVs under 3%, while Amazon reached 45.2%.

IOzone Performance Results The results of the IOzone benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 1%, but higher than Amazon by 6.4x and Rackspace by 6.4x. Internap SATA, Internap SSD had CVs under 2%, followed by Rackspace with 6.9% and Amazon with 13%.

0

5000

10000

15000

Ave

rag

e M

B/s

ec

RAM TEST - RAMSpeed SMP:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

20.0%

40.0%

60.0%Co

effi

cen

t o

f V

aria

tio

n

RAM TEST - RAMSpeed SMPPerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE

0

2000

4000

6000

Ave

rag

e M

B/s

ec

DISK TEST - IOzone:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

5.0%

10.0%

15.0%Co

effi

cen

t o

f V

aria

tio

n

DISK TEST - IOzonePerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE


24

MongoPerf – Write w/ Cache Performance Results The results of the Write with Cache benchmark test show the Internap SATA instance matching Amazon, but performing lower than the Internap SSD instance and Rackspace. The Internap SATA instance performed lower than the Amazon by 35%, the Internap SSD instance by 95% and Rackspace by 93%. Internap SSD had the lowest CV with 1.1%, followed by Amazon with 24.5%, Rackspace with 60% and Internap SATA with 92.5%.

MongoPerf – Read w/ Cache Performance Results The results of the Read with Cache benchmark test show the Internap SATA instance matching the Internap SSD instance, while performing lower than Amazon and higher than Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 1%, but lower than Amazon by 33% and higher than Rackspace by 4x. Internap SATA and Internap SSD had CVs under 2%, followed by Rackspace with 8.8% and Amazon with 31.9%.

0

100000

200000

300000

Ave

rag

e O

ps/

sec

DISK TEST - Write Ops (w/ Cache):Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

50.0%

100.0%Co

effi

cen

t o

f V

aria

tio

n

DISK TEST - Write Ops (w/ Cache)Performance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE

0

500000

1000000

1500000

2000000

Ave

rag

e O

ps/

sec

DISK TEST - Read Ops (w/ Cache):Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

10.0%

20.0%

30.0%

40.0%Co

effi

cen

t o

f V

aria

tio

n

DISK TEST - Read Ops (w/ Cache)Performance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE


25

MongoPerf – Write w/out Cache Performance Results The results of the Write without Cache benchmark test show the Internap SATA instance performing lower than the Internap SSD instance, Amazon and Rackspace. The Internap SATA instance performed lower than the Internap SSD instance by 94%, Amazon by 88% and Rackspace by 91%. Internap SSD had the lowest CV with 0.5%, followed by Internap SATA with 4.5%, Rackspace with 7.1% and Amazon with 9.5%.

MongoPerf – Read w/out Performance Results The results of the Read without Cache benchmark test show the Internap SATA instance performing lower than the Internap SSD instance, Amazon and Rackspace. The Internap SATA instance performed lower than the Internap SSD instance by 99%, Amazon by 95% and Rackspace by 60%. Internap SATA had the lowest CV with 5.4%, followed by Rackspace 17.6%, Internap SSD with 22.3% and Amazon with 75.6%.

0

1000

2000

3000

Ave

rag

e O

ps/

sec

DISK TEST - Write Ops (w/out Cache):Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

5.0%

10.0%Co

effi

cen

t o

f V

aria

tio

n

DISK TEST - Write Ops (w/out Cache)Performance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE

0

10000

20000

30000

Ave

rag

e O

ps/

sec

DISK TEST - Read Ops (w/out Cache):Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

20.0%

40.0%

60.0%

80.0%Co

effi

cen

t o

f V

aria

tio

n

DISK TEST - Read Ops (w/out Cache)Performance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE


26

Iperf Performance Results The results of Iperf benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 1%, but higher than Amazon by 3.2x and Rackspace by 3.2x. Internap SATA, Internap SSD and Rackspace had CVs less than 1%, while Amazon was slightly higher at 1.5%.

Ping Performance Results The results of the Ping benchmark test show the Internap SATA instance matching the Internap SSD instance, while outperforming Amazon and Rackspace. The Internap SATA instance performance differed from the Internap SSD instance by less than 1%, but higher than Amazon by 87% and Rackspace by 80%. Internap SATA, Internap SSD had CVs less than 1%, while Rackspace was 36.4% and Amazon was 22.2%.

0

500

1000

Ave

rag

e M

ibit

s/se

c

NETWORK TEST - Iperf:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

0.5%

1.0%

1.5%

2.0%Co

effi

cen

t o

f V

aria

tio

n

NETWORK TEST - IperfPerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE

0

1000

2000

3000

Ave

rag

e M

icro

seco

nd

s

NETWORK TEST - Ping:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

10.0%

20.0%

30.0%

40.0%Co

effi

cen

t o

f V

aria

tio

n

NETWORK TEST - PingPerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE


27

MySQL Performance Results The results of the MySQL Performance benchmark test show the Internap SATA instance performing lower than the Internap SSD instance, while slightly outperforming Amazon and Rackspace. The Internap SATA instance performance was lower than the Internap SSD instance by 2.6x, but higher than Amazon by 8% and Rackspace by 14%. Internap SATA had the lowest CV with 6.2%, closely followed by Rackspace with 6.3%, and then Internap SSD with 9.5% and Amazon with 12%.

0

5

10

15

20

Ave

rag

e S

eco

nd

s

DATABASE TEST- MySQL Performance:Internap SSD vs Internap SATA vs Amazon vs

Rackspace

SSD

SATA

AMAZON

RACKSPACE

0.0%

5.0%

10.0%

15.0%Co

effi

cen

t o

f V

aria

tio

n

DATABASE TEST - MySQL PerformancePerformance Stability of SSD vs SATA vs Amazon

vs Rackspace

SSD

SATA

AMAZON

RACKSPACE


28

Appendix B

Methodology

Server Setup: Two servers are set up on each provider that match the following configurations as closely as possible. The Primary Server runs all tests specific to CPU and storage. The Secondary Server is used in conjunction with the Primary Server to test the internal network.

Primary Server 4 vCPU 8 GB RAM 50 GB Storage

Secondary Server 1 vCPU 2 GB RAM 50 GB Disk

The following server configurations were set up on each provider. The pre-configured VMs that characterize Amazon and Rackspace offerings make it impossible to match target configurations exactly. Amazon Web Services EC2 http://aws.amazon.com/ec2/instance-types/ Primary: Data Center: Availability Zone East-1a OS: Ubuntu 12.04 vCPUs: 2 (with 4 ECU) RAM: 7.5GB Disk: 50GB (EBS) Secondary: Data Center: Availability Zone East-1a OS: Ubuntu 12.04 vCPUs: 1 (with 1 ECU) RAM: 1.7GB Disk: 160GB

Internap AgileSERVER bare-metal cloud http://www.internap.com/agile/flexible-cloud-hosting-solutions/enterprise-public-cloud-solutions/bare-metal-cloud/ Primary: Data Center: Santa Clara OS: Ubuntu 12.04 vCPUs: 4 RAM: 8GB Disk: 120GB SSD (and 2TB SATA) Secondary: Data Center: Santa Clara OS: Ubuntu 12.04 vCPUs: 1 RAM: 2GB Disk: 50GB

Rackspace OpenCloud http://www.RACKSPACE.com/cloud/servers/pricing/ Primary: Data Center: Dallas Fort Worth (DFW) OS: Ubuntu 12.04 vCPUs: 4 RAM: 8GB Disk: 50GB (Block) Secondary: Data center Dallas Fort Worth (DFW) OS: Ubuntu 12.04 vCPUs: 2 RAM: 2GB Disk: 80GB

Tests Used The benchmarks used to test the providers listed in this report are derived from a number of sources. CPU Internal network RAM Storage Database File Compression Iperf General RAM Test Iozone MySQL Audio Compression Ping Read w/ Cache Video Encoding Write w/Cache

Read w/out Cache Write w/out Cache

Timeframe The test period for the systems ranged from 9/26/2013 to 10/5/2013. Data Collection Performance data for each test was collected three times per day, every day throughout the above testing periods. Data shown in the graphs and tables only take the highest and lowest point of each day into account. Rather than showing all the data points collected, only showing the highest and lowest points of the day let customers determine the range of best to worst performance they could expect from a virtual server from the providers. Cloud Spectator obtains cloud servers by purchasing the server space directly from the providers as any user would. For certain providers, the client may reimburse Cloud Spectator for the sever space needed for data collection relevant to that

Cloud Spectator Performance Report: Internap, AMAZON and RACKSPACE September 2013

29

active project. Cloud Spectator collects and compiles the data into the CloudSpecs database and translates it into a visual display. Terms and Definitions For the purpose of understanding the relational values of the data, all numbers within the tables below each graph are expressed as whole numbers except the percentages, which are expressed to the tenth decimal point. Percentages are expressed in that manner to account for instances when the coefficient of variation (CV), which is expressed as a percentage, falls below 1%, indicating a high degree of performance predictability. Average When describing averages, Cloud Spectator refers to the average numerical value over a period of 10 days from 9/26/13 until 10/5/13. Average provider scores can be found inside the tables underneath each graph within this document. The average is used to summarize the data from the charts in a simplified overview. Standard Deviation The standard deviation is calculated over a period of 10 days from 9/26/13 until 10/5/13. The standard deviation can be found inside the tables underneath each graph within this document. The standard deviation is used to understand the amount of variation from the average benchmark score of a provider; i.e., how predictable a provider’s server performance is for that test. The standard deviation can only be used to understand the amount of variation within a certain provider, and cannot be used to compare among providers. Coefficient of Variation (CV) The coefficient of variation is expressed as a percentage. The CV can be found inside the tables underneath each graph within this document. The CV is a measure of precision. It normalizes the standard deviation as a percentage of the average, which can be compared across providers. A lower CV means more stable performance.

[(Standard Deviation) / (Average)] * 100 10-Day Highs and Lows From the tested period between 9/26/13 until 10/5/13, Cloud Spectator extracts and presents the highest and lowest achieved scores by each provider in the tables underneath the graphs within this document.

Cloud Spectator Performance Report: Internap, AMAZON and RACKSPACE September 2013

30

About Cloud Spectator Cloud Spectator is the premier, international cloud analyst group focused on infrastructure pricing and server performance. Since 2011, Cloud Spectator has monitored the cloud Infrastructure industry on a global scale and continues to produce research reports for businesses to make informed purchase decisions by leveraging its CloudSpecs utility, an application that automates live server performance tests 3 times a day, 365 days a year with use of open source benchmark tests. Currently, the CloudSpecs system actively tracks 20 of the top IaaS providers around the world. Cloud Spectator 800 Boylston Street, 16th Floor Boston, MA 02199 Website: www.cloudspectator.com Phone: (USA) 1-617-300-0711

performance analysis: benchmarks of bare-metal & virtual ... · performance testing and...

Documents