comparison of solid state drives (ssds) on different … › wp-content › uploads › 2016 › 02...

COMPARISON OF SOLID

STATE DRIVES (SSDS) ON

DIFFERENT BUS INTERFACES Technical Report

ABSTRACT In this report, we discuss the results of an

experiment comparing solid state drives (SSDs)

attached to different system bus interfaces. TPC-H

and TPC-C benchmarks have been executed to

compare NVMe and SATA SSDs. The results indicate

that modern protocols like NVMe over PCIe can

overcome the negative effect traditional disk

controllers have on performance, latency, and

bandwidth. Tests were conducted for different

system configurations and various data parameters

were collected, including energy and power.

Antonio Cisternino, Maurizio Davini TR-01-15

IT Center Università di Pisa P.I.: 00286820501 C.F.: 80003670504 Largo Bruno Pontecorvo, 3, I-56127 Pisa Italy

*Other names and brands may be claimed as the property of others.

Executive Summary In this paper we present the results of a set of experiments made to compare the performance and the differences

between Intel SSD drives (20nm MLC) with SATA and NVMe PCIe® interfaces. The workload used for the

experiment is the HammerDB (DBMS) on Windows Server* 2012 R2 with Microsoft SQL Server* 2014 Enterprise.

TPC-C and TPC-H benchmark tests were run on the two drive types while varying the memory available to the

DBMS. Experiment results demonstrate how system software effectively used the superior bandwidth of the SSDs

using the NVMe protocol over PCIe to deliver significant performance improvements on intensive Disk I/O

workloads. We recorded a 2x speedup factor running the TPC-H query set when the database was stored on the

NVMe PCIe SSD, leading to one-half the completion time compared to storing the database on the SATA SSD.

Moreover, there was significant improvement on the latency in accessing data with NVMe PCIe interface versus

SATA: we found that in any situation the disk queue length (i.e. the number of requests delayed because a drive

is busy) was greater for SATA drives by a factor of 10 in many cases, even when Disk I/O was small. Given that both

drives were manufactured by Intel using the same technology, we conclude that the NVMe protocol over PCIe

offers significant advantages over the older SATA controller, not only with high throughput workloads but also for

low latency, sensible workloads found in widespread reactive systems like web servers and virtualization.

Introduction Hardware technology and capability seem to improve at a much faster pace than software, making it a serious

challenge to test hardware devices for specific workloads. The advent of solid state drives (SSDs) has disrupted

the long-standing assumptions about latency and transfer speed of persistent storage made by many

programmers. In fact, SSD technology has advanced to the point that (a) the disk controller is becoming the bottle

neck of the communication bandwidth; and (b) rotating disks are the bête noire of latency-intolerant applications

like content delivery. Simply switching to SSDs can improve application performance and lower latency vs.

standard rotating disk. To overcome the bottleneck of the disk controller, SSD storage is available in the PCIe card

form factor. However, the multipurpose PCIe bus has a number of limitations when it comes to stored data

manipulation, the most significant of which is the lack of specialized commands to program the bus for specific

storage uses to minimize interoperability problems. For this reason, the storage-specific Non-Volatile Memory

Express (NVMe) protocol was defined. According to its authors, NVM Express is an optimized, high performance,

scalable host controller interface with a streamlined register interface and command set designed for Enterprise,

Datacenter, and Client systems that use Non-Volatile Memory storage [1]. NVMe is quickly emerging as the

standard for attaching SSD drives directly to the PCIe bus for improved application performance and bandwidth

(at lower latency) while retaining the convenient disk drive form factor.

In this technical report, we present experiment results comparing Intel® SSD Data Center S3610 Series (“S3610”,

1.6TB, 2.5in SATA 6Gbps, 20nm, MLC) [2] attached to a HBA/bus controller on the SATA bus and Intel® SSD Data

Center P3600 Series (“P3600”, 1.6TB, 2.5in PCIe 3.0 (8Gbps), 20nm, MLC) [3] –using the NVMe protocol– directly

connected to the PCIe bus (no separate HBA/adapter required). Tests were conducted on a NVMe-capable Dell*

PowerEdge* R630 rack server with 2 Intel® Xeon® E5-2683 v3 processors and 128GB of RAM. The workload used

is HammerDB; in particular, we performed TPC-C benchmark (modeling OLTP) and TPC-H benchmark (modeling

decision support system) using Microsoft* SQL Server* 2014 Enterprise edition as the database manager running

Microsoft* Windows Server* 2012 R2 Standard edition operating system. The specific details about the numbers

obtained, system configuration, and testing procedures are discussed in the appendix.



General Considerations After installing Windows Server* 2012 R2 with all the required drivers, including Intel drivers for the NVMe drives,

we used CrystalDiskMark* v5.0.2 x64 to test the SATA and NVMe SSDs to confirm that the bandwidth and IOPS

were matching the expected performance. The default CrystalDiskMark setting runs five iterations of four test

configurations: 1) sequential 128KiB (131.1KB) block size transfers with 32 queues, 1 thread – read and write;

2) random 4KiB (4.1KB) block size transfers with 32 queues, 1 thread – read and write; 3) sequential 1MiB (1.05MB)

block size transfers with 1 queue, 1 thread – read and write; 4) random 4KiB (4.1KB) block size transfers with 1

queue, 1 thread – read and write. The largest test file size supported by CrystalDiskMark, 32GiB (34.3KB), was

chosen to maximize the amount of disk IO, as denoted by the 66% disk utilization (Note: CrystalDiskMark uses

binary byte notation – 1KiB=1024B (vs. decimal byte notation where 1KB=1000B)).

The baseline result showed the peak sequential read performance of NVMe SSDs was more than five times faster

than SATA SSDs; the peak sequential write performance was more than three times faster with NVMe SSDs. Peak

random read/write performance was more than 2.5x/3x higher with NVMe SSDs. NVMe SSDs even outperformed

SATA SSDs when comparing read/write throughput with single queue/single thread (amazingly, both drives have

very similar low queue depth random read specification, hence the nearly identical performance; we suspect this

is due to the lack of parallelism at 1 queue/1 thread). These results suggest NVMe SSDs would be well-suited for

high throughput workloads.

However, benchmarks like HammerDB* are more representative of real-world, high throughput workloads such

as databases. Therefore, we installed HammerDB and configured the server according to the various online

guidelines from the HammerDB and Microsoft web sitesii. HammerDB comes with two benchmarks for simulating

two different environments: TPC-C simulates a warehouse management as a typical example of elevated number

of transactions; TPC-H simulates decision support system type of queries where long computations and complex

queries are needed to obtain results.

HammerDB is a tricky benchmark to set up, offering many options to control the workload. In this technical report,

we mainly focus on TPC-H output, which offers more interesting results than TPC-C (more information about TPC-

C is provided in the appendix). TPC-H uses a script (with possible scale factors of 1, 10, 30, 100, 300, or 1,000) to

generate databases of different sizes. We chose a scale factor of 100 (this value is due to preliminary tests

discussed in the appendix), resulting in a 300GB database (220GB for the .mdf file and 80GB for the .log file). We

altered the generated database to use clustered indexes as suggested by several online documents.

1.6TB Intel® SSD DC P3600 (NVMe) 1.6TB Intel® SSD DC S3610 (SATA)



HammerDB testing in Microsoft SQL Server is mainly governed by two

parameters: the maximum degree of parallelism for queries (MAXDOP)

and the number of virtual users that concurrently execute the 22 queries

defining the benchmark. We set MAXDOP to 56 to match the number of

hyper-threads of the machine (each CPU has 14 cores). We tested several

values and found that the best use of system resources was with 16 virtual

users; we decided to test 1, 8, and 16 virtual users.

In any experiment, the crucial point is the question that you have in mind

before reading the results. In our case, the point was to understand the

different behavior of SATA SSD drives versus NVMe SSD drives. In many

documents about disk drive benchmark, the peak throughput seemed to

be the only interesting parameter. For SQL Server the customary

suggestion was to restrict the memory used by the DBMS in order to force

more disk I/O to the database. Another strong suggestion was to use an

amount of RAM equivalent to 10% of the database size as a way to simulate

intensive I/O DBMS conditions. We decided to iterate the TPC-H

benchmark for different DBMS memory size constraints in order to

understand the disk performance under different conditions.

Every combination of RAM size and number of virtual users was monitored

using Windows performance monitor. For every relevant disk in the system

(boot and storage, both SATA and NVMe) we recorded these details.

Disk-related parameters:

- Disk Read Bytes/second

- Disk Write Bytes/second

- Disk Bytes/second

- Current Disk Queue Length

CPU-related parameters:

- Core percentage use (0..53)

- % Idle time, % Interrupt time

- % User time

- % Processor time

- % Privileged time

Energy-related parameters:

- Power

Database-related parameters:

- Total server memory (KB)

- Transactions/second

- Write transactions/second

The idea of a “cluster” is to tune

the database table entries to the

performance of a particular

index value. Basically, the

records of the table will be

reordered on the disk

(clustered) to match the order

of the specified index. This has

the benefit of rapid scan and

retrieval of records falling into

some range of clustered index

values. There can, therefore, be

only one clustered index.

Writing to a table with a

clustered index can be slower,

especially if the data needs to be

rearranged.

With a non-clustered index

there is a logical order that does

not match the physical order on

disk (the index is aligned to the

logical pointer instead of the

physical disk). You can have

many non-clustered indexes.

Index and update operations are

faster with non-clustered index;

however each new index

increases the time it takes to

write new records.

CLUSTERED VS. NON-CLUSTERED INDEX



We also recorded the completion time for each benchmark configuration. In general, NVMe drives performed

better than SATA drives as expected. What was unexpected for us was the huge difference, even when disk I/O

was small. We used 3D representations for showing data because trends are more important than numbers, and

such representation express them better in our opinion than multiple lines.

SATA Drive Performance Results In the figures below, you can observe the completion time, average and maximum disk throughput and the

average disk queue length for the various configuration of TPC-H tested. SQL Server was capable of effectively

using the disk throughput as you can notice the maximum disk transfer that is always above 500MB/s (theoretical

interface limit is 600MB/s).

However, costraining the amount of DBMS memory reduced the in-memory cache, increased disk usage, and

changed the database performance. When memory size was limited to 16GB or less, the completion time grew

from less than 1-hour to nearly 3-hours and average disk transfer (I/O throughput) was reduced by as much as

44% from the max throughput obseved. The effect was more pronounced with more virtual users executing

queries in parallel. The one virtual user case was taken to establish a baseline but it could not saturate the system

resources other than the CPU, which consistently showed ~90% utilization.

A particular interesting metric was the average disk queue length, which is a proxy for disk latency. It was surprising

that even with unlimited memory and very few I/O bursts, the average queue length was in the 3-5 range (meaning

disk fetches were still queued, resulting in idle system resources (see “Disk Queue (SATA/NVMe)” section below

for the table with recorded values). It is natural and expected that queue length increases with I/O, but we did not

expect such high values for low I/O throughput. We believe this was due to some form of buffering of the SATA

controller, perhaps a reasonable I/O wait assumed for mechanical drives.

1

160

5000

10000

4Gb 8Gb16Gb

32GbUnlim

Completion time (s)

1

160,00

100,00

200,00

300,00

400,00

4Gb 8Gb16Gb

32GbUnlim

Avg Disk MB/s

1

16525,00530,00

535,00

540,00

545,00

4Gb 8Gb16Gb

32GbUnlim

Max Disk MB/s

1

160,00

50,00

100,00

4Gb 8Gb16Gb

32GbUnlim

Avg Queue length



In conclusion, running a 300GB DBMS (TPC-H) on SATA SSDs was able to saturate the disk bandwidth during write

bursts; however longer completion times (especially with smaller memory footprint) and higher disk latency (even

at low I/O throughput) would result in lower database performance, most appreciably as the number of virtual

users and/or databse size grew.

NVMe Drive Performance Results At first glance, the four NVMe graphs appeared to have a similar shape as the SATA drives; however a closer look

revealed some significant differences. The peak NVMe SSD completion time was about 1-hour less than with SATA

SSDs; in both cases, the peak time occurred with 16 virtual users and 8GB memory size (this is because some

queries failed to complete with 4GB of memory due to insufficient resources).

With NVMe SSDs, the DBMS was able to achieve 2GB/s maximum disk I/O transfers with 16 virtual users and DRAM

memory constrained to 16GB. The NVMe SSDs were able to sustain this throughput even with a svelte 4GB DRAM

memory footprint. The shape of the NVMe average I/O graph demonstrated that even when the DBMS thrashes,

it can reach up to 800MB/s disk transfers (which is more than the 500MB/s recorded for SATA drives and close to

the PCIe interface theoretical limit of 985MB/s).

If the difference of scale for disk throughput was significant between SATA and NVMe SSDs, the queue length was

even moretelling. With unconstrained memory the average queue length was almost 0 (meaning virtually no disk

latency and system resources were not idling waiting for data transfers from disk). Even when the I/O traffic on

the drive increased, the queue length remained smaller than that observed with SATA drives. The fact that the

two drive types were manufactured using the same solid state technology suggests the SATA controller has

become the bottle neck of the drive pipeline.

1

160

2000

4000

6000

4GB8GB

16GB32GB

Unlim

Completion time (s)

1

160,00

200,00400,00600,00800,00

4GB 8GB16GB

32GBUnlim

Avg Disk MB/s

1

160,00

1000,00

2000,00

3000,00

4GB 8GB16GB

32GBUnlim

Max Disk MB/s

1

160,00

10,00

20,00

30,00

4GB 8GB16GB

32GBUnlim

Avg Queue length



With these results for our DBMS running on NVMe SSDs, we reaached the following conclusions:

- A DBMS can reduce the negative performance effect of disk thrash by using NVMe PCIe SSDs to increase

available bandwith;

- The latency of NVMe PCIe drives is significantly smaller than historical assumptions, leading to faster disk

access and data retrieval;

- NVMe PCIe SSD bandwidth saturation occurs later than SATA SSDs or traditional drives, allowing for more

responsive systems with limited system resources.

SATA vs. NVMe comparison In order to compare the two datasets recorded for SATA SSDs and NVMe SSDs, we calculated the ratio between

SATA numbers and NVMe numbers obtained for each measure. In this section we discuss the ratio of the various

parameters already presented as well as the energy consumption recorded during the tests.

Completion Time (SATA/NVMe) The TPC-H benchmark executes 22 queries for every virtual user according to the maximum degree of parallelism

defined in the test configuration. For both SATA and NVMe tests, when the DBMS memory was constrained to

4GB some of the queries failed to complete due to insufficient resources.

Completion time ratio (s) Virtual users

Memory 1 8 16

4GB 1,65 2,09 1,94

8GB 1,20 1,79 1,53

16GB 1,04 1,06 1,05

32GB 1,15 1,00 0,96

Unlim 1,15 1,01 1,00

1

8

160,00

0,50

1,00

1,50

2,00

2,50

4GB8GB

16GB32GB

Unlim

1

8

16



The completion time favors NVMe as indicated by a ratio value almost always greater than 1; however, to get real

benefit from the superior I/O throughput offered by NVMe PCIe drives, it is necessary to run a workload that

produces continuous I/O rather than just I/O spikes as a DBMS with abundant memory does. When I/O is sustained

then completion time is affected (longer). When the SATA drive saturates the bus, the host/system can realize up

to 2x speedup by switching to NVMe PCIe drives.

NVMe PCIe SSDs provides faster completion time compared to SATA SSDs and this is accentuated with continuous I/O or limited memory in the system. This overall means that tasks can be quicker finished, as well as the needed memory, versus

using SATA, can be reduced, which will reduce the overall memory costs.

Disk throughput (SATA/NVMe) Disk throughput is often considered a key factor for application performance, but as available bandwidth increases

it is legitimate to wonder if software is capable of benefitting from it. Our test results clearly demonstrate that 1)

the DBMS is capable of using the full bus bandwidth if available, and 2) the main reason for the reduction in

completion time of the benchmark test was the higher bus bandwidth of NVMe PCIe drives over SATA.

Avg Disk transfer ratio Virtual users

Memory 1 8 16

4GB 0,64 0,44 0,49

8GB 0,83 0,69 0,70

16GB 0,92 1,04 0,87

32GB 0,88 1,23 1,04

Unlim 0,86 0,99 0,98

In this case, the ratio of the average disk bandwidth is typically less than 1 because the SATA average bandwidth

is smaller than the NVMe PCIe drive. Reviewing the appendix for the two drive types, we can see that the average

disk bandwidth ratio can be as low as 0.23, which is close to the 0.44 ratio value recorded.

NVMe PCIe SSDs delivers higher Disk Throughput, meaning that DBMS applications are able to use the available bandwidth and are no longer as constrained from it.

1

8

160,00

0,20

0,40

0,60

0,80

1,00

1,20

1,40

4GB8GB

16GB32GB

Unlim

1

8

16



Disk Queue (SATA/NVMe) As stated in the specs and confirmed by the I/O benchmark, disk bandwidth favored the NVMe SSDs. Furthermore,

we confirmed that real applications such as DBMS under the DSS-like simulated workload of TPC-H is able to

benefit from the increased bandwidth.

Modern workloads are often acutely affected by a subtler parameter than sheer bandwidth: latency! For

mechanical disks, seek time was on a scale that any other contribution to latency was insignificant. Disk controllers

use caching and reordering techniques to reduce seek operations, but such techniques are useless when an SSD

drive is used in place of a mechanical one.

Latency is related to disk operation requests that the operating system stores (queues) when a drive is busy. If

requests arrive at a faster rate than the drive can handle, then the length of the queue grows (up to 255 in

Windows*).

Avg Queue length ratio Virtual users

Memory 1 8 16

4GB 3,36 3,47 3,38

8GB 4,71 4,74 5,08

16GB 5,49 8,63 7,40

32GB 10,89 23,70 10,91

Unlim 7,72 26,44 35,55

While we were surprised by the average disk queue length of SATA compared to NVMe drives, the ratio between

the two drives was unusually large. The largest difference was observed with unconstrained (unlimited) DRAM,

although the 35x factor (i.e. the queue is 35 times longer for SATA than NVMe on the same workload) is most

likely a calculation aberration due to the near-zero average queue length for NVMe drives. When the DRAM

memory is constrained, a 10x factor is found; even under heavy I/O workload with 4GB DRAM, the ratio is >3x.

1

8

160,00

5,00

10,00

15,00

20,00

25,00

30,00

35,00

40,00

4GB8GB

16GB32GB

Unlim

1

8

16



From these numbers we conclude that NVMe SSD drives offer a huge advantage over SATA SSD drives under all

test conditions. Lower latency NVMe SSDs offer better performance for many applications, and in particular, to

Web applications where disk access dominates the overall request handling time. We also believe that in a

virtualization scenario, especially for interactive virtual machines, the lower latency of NVMe drives can

significantly improve the user experience.

NVMe PCIe SSDs are able to handle many times more requests than SATA SSDs which results that applications sensitive to latency will be seen much benefits by

using NVMe PCIe SSDs

Energy Computing efficiency is a critical factor nowadays, and we considered the power profile of the benchmarks run as

monitored through the Windows performance counter.

The profile of power usage shows correlation with CPU and disk usage during the test. As an example consider the

following profile obtained from the NVMe benchmark for 16 users and 32GB of RAM:

When increased disk I/O caused a CPU slowdown, power followed the CPU trend (but remained higher

than the idle power state);

The idle system consumed 254W; the maximum value with disk I/O was 348W and the minimum was

~300W. in other words, CPU usage dominated the power budget whereas disk power profile was roughly

50% of the CPU (marginal, bordering on irrelevant, impact on the overall result).

We estimated the energy required for each benchmark by multiplying the average power by the completion

time. As we can observe the shape of the completion time ratio and the energy one are similar; however, if

we look at the values we can notice that the energy ratio between SATA and NVMe is slightly smaller than the

completion time ratio. This is due to the fact that when disk I/O dominates, the CPU efficiency in managing

energy reduces the overhead of the idle CPU when waiting for disk operations. In fact, if we look to the average

CPU usage ratio we can see how the CPU is less used leading to a larger energy overhead.



Energy ratio Virtual users

Memory 1 8 16

4GB 1,56 1,93 1,81

8GB 1,17 1,71 1,48

16GB 1,03 1,06 1,05

32GB 1,12 1,00 0,97

Unlim 1,12 1,01 1,00

CPU ratio Virtual users

Memory 1 8 16

4GB 1,53 2,05 2,00

8GB 1,22 1,33 1,25

16GB 1,21 1,12 1,08

32GB 1,19 0,98 0,96

Unlim 1,17 1,02 0,94

Disk I/O completion time dominated CPU utilization and energy consumption. Disk drives have a similar power

profile, though the ability of NVMe PCIe SSD to achieve shorter completion times may provide significant energy

savings; in our tests, NVMe PCIe drives used almost 50% less energy than equivalent SATA SSDs.

By using NVMe PCIe SSDs you are able to reduce the overall disk power consumption due to tasks finishing more quickly, which reduces

overall system power usage.

1

160,00

1,00

2,00

3,00

4GB 8GB 16GB 32GB Unlim

Completion time ratio (SATA/NVMe)

1

8

16

1

160,00

0,50

1,00

1,50

2,00


Energy ratio (SATA/NVMe)

1

8

16

1

160,00

0,50

1,00

1,50


CPU ratio (SATA/NVMe)

1

8

16



Conclusions In this report we presented the results of our TPC-H benchmark tests comparing NVMe PCIe and SATA SSD drives.

The benchmark was executed on a server with Intel SSD DC S3610 Series (SATA), Intel SSD DC P3600 Series (PCIe)

and Microsoft SQL Server 2014 Enterprise on Windows Server 2012 R2 Standard.

The test results confirmed that:

- NVMe SSDs can provide more than 4x increase in usable bandwidth to real applications such as DBMS;

- NVMe SSDs offer a 4x to 35x latency reduction under any I/O condition;

- The energy consumption for a workload is reduced by 50%, compared to SATA, because of the improved

performance of NVMe SSDs.

All the disks tested were manufactured using the same SSD flash memory technology to highlight the limitation of

SATA controller with respect to NVMe over PCIe protocol.

Bibliography 1. http://www.nvmexpress.org/

2. http://ark.intel.com/products/82934/Intel-SSD-DC-S3610-Series-1_6TB-2_5in-SATA-6Gbs-20nm-MLC

3. http://ark.intel.com/products/80992/Intel-SSD-DC-P3600-Series-1_6TB-12-Height-PCIe-3_0-20nm-MLC

4. http://ark.intel.com/products/81055/Intel-Xeon-Processor-E5-2683-v3-35M-Cache-2_00-GHz

5. http://www.hammerdb.com/hammerdb_mssql_oltp_best_practice.pdf

http://www.nvmexpress.org/

http://ark.intel.com/products/82934/Intel-SSD-DC-S3610-Series-1_6TB-2_5in-SATA-6Gbs-20nm-MLC

http://ark.intel.com/products/80992/Intel-SSD-DC-P3600-Series-1_6TB-12-Height-PCIe-3_0-20nm-MLC

http://ark.intel.com/products/81055/Intel-Xeon-Processor-E5-2683-v3-35M-Cache-2_00-GHz

http://www.hammerdb.com/hammerdb_mssql_oltp_best_practice.pdf



Appendix

Hardware Configuration Tests have been performed on a Dell* R630 with the following configuration:

- Dell R630 with support for up to four NVMe drives

- Perc* H730 Mini controller for Boot drive

- 2 Intel® Xeon® E5-2683 v3 2GHz CPU

- 2 Intel® SSD DC S3710 Series (SATA boot drive)

- 4 Intel® SSD SC P3600 Series (NVMe PCIe)

- 4 Intel® SSD DC S3610 Series (SATA)

- 128Gb (8x16Gb) DDR4 RAM

System Setup The system has been installed with Windows Server 2012 R2 Standard edition with the drivers provided by Dell

for the R630 and the Intel® Solid State Drive Data Center Family for NVMe Drivers. We then installed Microsoft

SQL Server 2014 Enterprise edition and all the latest updates from Microsoft. The SATA RAID hardware control for

Intel SSD DC S3610 Series has been configured in pass-through mode. The system has joined the AD of our

network.

After preparing the system we installed CrystalDiskMark* v5.0.2 and HammerDB* v2.18.

HammerDB Configuration We created several databases using the build scripts provided by HammerDB for both TPC-C and TPC-H to ensure

that the benchmark was performing well on the server after the configuration. The goal was to find the proper

arguments for DB creation capable of loading CPU and I/O without trashing the benchmark performance.

After several attempts we created the databases shown in the following figure using 56 as concurrent users and

max degree of parallelism:

TPC-C databases have been created as follows1:

Name #Warehouses

tpcc2 166

tpcc4 505

tpcc3 5000

1 Notice that we used suffixes on DB files and names to allow DBMS to mount all of them simultaneously. We use the prefix to indicate the DB for a particular benchmark.



We found that a scale factor of 100 for TPC-H database was able to generate a database big enough to stress disk

I/O keeping benchmark completion times acceptable. We generated and tested databases with larger and smaller

scale factors and obtained similar performances as the benchmarks discussed in this paper.

Once generated the databases we used the copy feature of SQL Server 2014 to copy the databases from NVMe

drives to the SATA ones. At the end of the replication process we obtained two sets of identical databases we used

for executing the tests. Both .mdf and .log files were stored on the same drive.

For TPC-H database we created cluster indexes on a different NVMe drive and we used the same drive for database

stored on SATA and NVMe drive. This choice has been made to ensure that the workload of the TPC-H was on the

database access rather than access to the indexes. Since the cluster indexes were stored on a NVMe drive the

discrepancy found between the two drives is considered to be even more significant.

Test Methodology Once the configuration had been validated we executed TPC-C and TPC-H benchmarks using the following

procedure:

- Configure the memory constraints on SQL Server

- Restart SQL Server

- Start HammerDB

- Configure connection to the desired DB

- Configure virtual users

- Start perfmon and start recording performance counters

- Start TPC-H benchmark

- Record the test duration

- Save the performance counters files

We repeated the procedure against the same parameters several times to ensure that the performance was not

affected by cache hits. This test has validated the procedure above.

TPC-C General info The TPC-C benchmark has been executed on the server but it failed to stress the system enough to show the

difference between NVMe and SATA drives. This is the main reason for commenting TPC-C only in the appendix of

this document.

The only parameter we found relevant has been the average disk queue length, which has been significantly

smaller even with a workload with little spikes due to periodical I/O bursts. The following picture shows the disk

counters measured for database TPCC3 (NVMe on the left, SATA on the right).



Even if in this case the average is less significant due to the disk traffic shape, we recorded a 1.54x factor of the

SATA queue length versus the NVMe one.

TPC-C has performed similarly on the two drive types, though with the smallest DB we recorded more than 10x in

the transaction per minute value recorded by the benchmark. In some occasions we recorded up to 14Mtpm for

TPCC2, versus 900Ktpm for TPCC3 and TPCC4 databases.

We do not believe that the sheer numbers are relevant for our purpose, but we found interesting the tpm graph

profile as displayed by TPC-C interface shown below (notice that the graph do not represent the full test

execution).

As you can easily notice the SATA disk bottleneck on throughput leads to a tpm shape less regular with significant

dents with respect to NVMe disk. However, these slowdowns in the transaction per minute count are not enough

to significantly affect the TPC-C completion time. Nevertheless, they correspond to sequential log writes that

follow the checkpoint needed to sync dirty buffers to disk, and the NVMe disk reduce the time required for syncing,

Database: TPCC2 Disk: NVMe



Database: TPCC2 Disk: SATA





improving the responsiveness of the DBMS. More information about configuring SQL Server for TPC-C with more

considerations about this phenomenon is available in bibliography [5].

Also for TPC-C we found that the disk queue length is shorter for NVMe, confirming smaller latencies in accessing

the drive.

SATA Test Tables In this section we report the tables whose graphs are shown in the SATA evaluation section for the TPC-H

benchmark.

Avg Disk MB/s Virtual users

Memory 1 8 16

4GB 272,9585724 343,81 344,41

8GB 154,4063625 275,5 268,21

16GB 166,2208681 343,54 304,67

32GB 137,5793819 124,92 138,12

Unlim 142,1327839 27,083 14,036

Max Disk MB/s Virtual users

Memory 1 8 16

4Gb 535,13095 536,84 540,66

8GB 540,9601154 541,61 541,92

16GB 534,2650928 538,78 544,8

32GB 533,9929695 538,12 539,13

Unlim 535,6773767 534,7 532,97

Completion time (s) Virtual users

Memory 1 8 16

4GB 479 3765 7376

8GB 709 4034 8182

16GB 302 1818 3691

32GB 142 879 1749

Unlim 131 694 1347

Avg Queue length Virtual users

Memory 1 8 16

4GB 67,922 84,069 83,817

8GB 35,351 68,153 65,554

16GB 36,736 78,024 68,096

32GB 29,891 17,419 19,568

Unlim 22,911 5,553 3,271



NVMe test tables In this section we report the tables whose graphs are shown in the NVMe evaluation section for the TPC-H

benchmark.

Avg Disk MB/s Virtual users

Memory 1 8 16

4GB 423,2865486 789 708

8GB 186,0174398 401 384

16GB 181,1741476 331 349

32GB 156,136487 101 133

Unlim 164,4890966 27,4 14,3

Max MB/s Virtual users

Memory 1 8 16

4GB 2197,925117 2307 2299

8GB 2282,954904 2288 2290

16GB 2251,791666 2095 2243

32GB 1315,692552 1546 1533

Unlim 1625,651889 1069 1174

Completion time (s) Virtual users

Memory 1 8 16

4GB 290 1803 3797

8GB 593 2250 5353

16GB 291 1712 3508

32GB 123 881 1819

Unlim 114 685 1351

Avg Queue length Virtual users

Memory 1 8 16

4GB 20,197 24,2 24,8

8GB 7,511 14,4 12,9

16GB 6,687 9,04 9,2

32GB 2,746 0,74 1,79

Unlim 2,966 0,21 0,09



Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You

should visit the referenced web site and confirm whether referenced data are accurate.

Intel, the Intel logo, Intel® Xeon®, Intel® SSD DC S3610, Intel® SSD DC S3710, and Intel® SSD DC P3600

are trademarks of Intel Corporation in the U.S. and/or other countries.

http://www.intel.com/performance

comparison of solid state drives (ssds) on different … › wp-content › uploads › 2016 › 02...

Documents