© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Maximizing EC2 and Elastic Block Store Disk Performance

Todd Varland, Solutions Architect, AWS

What We’ll Cover- Maximizing EC2 and Elastic Block Store

Performance – Best Practices- As Measured by…- Configuration Options- Deployment Patterns- Tips and Best Practices

So.. what storage workloads?• Relational DB

• NoSQL DB

• Data warehousing

• Files

• Media

For most builders AWS is get in and go!

Other AWS choices we won’t cover

• Amazon RDS (managed RDBMS)

• Amazon DynamoDB (managed NoSQL)

• Amazon ElastiCache (in-memory caching service)

• Amazon Redshift (managed data warehouse)

• Amazon S3 (object store service)

• Amazon Glacier (archive storage service)

A “Normal” Hard Drive

EBS =

What is Amazon EBS?

• Very flexible service with lots of choice – Used with Amazon EC2 instances– Attach/detach/copy/delete volumes– Point-in-time snapshots of volumes -> Amazon S3– Automatically replicated within its Availability Zone to protect

from component failure– Paying a low price for only what you provision

Key Pieces

AmazonEC2

An I/O

AmazonEBS

Network link

Measured by

1. Throughput: Read/Write rate to storage (MB/s)

2. Latency: Delay between request and completion (ms)

3. Cost: How much to run this ($)

Tools available for tuning

1. EC2 Instance: Network transfer rate (Mbps)

2. EBS Optimized: EC2 instance option (On/Off)

3. PIOPS: Provisioned I/O operations per second (#)

4. Queue Depth: The number of outstanding I/Os (#)

5. Block Size: The size of each I/O (KB)

6. Workload: Block size, read/write ratio, serialization

Amazon EBS Standard

Amazon EBS

• IOPS: ~100 IOPS steady-state, with best-effort bursts to hundreds

• Throughput: variable by workload, best effort to 10s of MB/s.

• Latency: Varies, reads typically <20 ms writes typically <10 ms

• Capacity: As provisioned, up to 1 TB

Amazon EBS PIOPS

Amazon EBS

• IOPS: Within 10% of up to 4000 IOPS, 99.9% of a given year, as provisioned.

• Throughput: 16 KB per I/O = up to 64 MB/s, as provisioned.

• Latency: low and consistent. Second/IOPS, at recommended QD

• Capacity: As provisioned, up to 1 TB

**

EC2 Instance: Architecting for Performance

• IOPS consistency requires EBS-optimized instances

• Maximum throughput delivered by Amazon EBS is limited by Amazon EC2 bandwidth

• EBS throughput =

EBS IOPS × Block size – Ex: 64 MB/s = 4000 IOPS × 16 KB

Instance vCPUEBS

Optimized Max MB/s Max 16k IOPSt1 micro 1  No 32MB/s  2000m1.small 1  No  64MB/s 4000

m1.medium 1  No 64MB/s  4000 m1.large 2 Yes 64MB/s 4000m1.xlarge 4 Yes  128MB/s 8000 m3.xlarge 4 Yes 64MB/s 4000

m3.2xlarge 8 Yes  128MB/s 8000 c1.medium 2  No 32MB/s  2000c1.xlarge 8 Yes  128MB/s 8000 

cc2.8xlarge 32  NA  800MB/s 50,000 m2.xlarge 2  No  64MB/s 4000

m2.2xlarge 4 Yes 64MB/s 4000m2.4xlarge 8 Yes  128MB/s 8000 cr1.8xlarge 32  NA 800MB/s  50,000 hi1.4xlarge 16  NA 800MB/s  50,000 cg1.4xlarge 16  NA 800MB/s  50,000

Max 8k = 2x

Max 4k = 4x*

Max 2k = 8x*

*Maximum IOPS is also limited to ~100,000 per 32 vCpu, irrespective of block size/throughput.

EBS-Optimized• EBS-optimized offers a “SAN-like” experience

• Network interference results:

No impact on IOPS or Amazon EBS throughput

Row Labels AvgBW AvgIOPs

m3.2xlarge (EBS-optimized)

no network loadrandom    

read 57,542 3,596

write 61,713 3,857

rw (70/30) 66,997 4,186

sequential    

read 61,708 3,856

write 61,651 3,853

rw (70/30) 66,996 4,187

with network load-test1random    

read 59,835 3,739

write 63,407 3,962

rw (70/30) 68,859 4,303

sequential    

read 61,736 3,858

write 63,360 3,959

rw (70/30) 68,859 4,302

Network interference tests

No Difference

InThroughput

EC2 Instance

An I/O

AmazonEBS

• BandwidthJust because Amazon EC2 sends morework doesn’t mean there’s enough bandwidth to handle it!

AmazonEC2

EC2 Instance

An I/O

• Bandwidth Without more bandwidth, more Amazon EBS volumes or higher PIOPS won’t help!

AmazonEC2

❶ Select a new type of Provisioned IOPS volume

❸ Specify the number of I/O operations per second your application needs, up to 4000 IOPS per volume. The volume will deliver the specified I/O operations per second.

❷ Specify the volume capacity

Review: Provisioned IOPS Volumes

Minimum ratio of capacity to IOPS = 1:30

$aws ec2 create-volume --availability-zone us-east-1a --size 134 --volume-type io1 --iops 4000

EBS PIOPS

AmazonEC2

An I/O

AmazonEBS

16 KBytes

USE PIOPS + EBS-OPTIMIZEDPLEASE

I/O Characteristics• I/O size

– 4 KB to 64 MB

• I/O pattern– Sequential and random

• I/O type– Read and write

• PIOPS always measures I/O in terms of 16 KB or smaller

• PIOPS delivers same number of IOPS for sequential and random I/O

• PIOPS delivers same number of IOPS for reads or writes

PIOPS is optimized for database workloads

PIOPS

an I/O

Amazon EBS

• Insufficient PIOPS provisioned

Just because Amazon EC2 sends more work, doesn’t mean there’s enough IOPS to handle it!

AmazonEC2

PIOPS

AmazonEC2

An I/O

AmazonEBS

• Insufficient PIOPS provisioned

Adding bandwidth alone or increasing instance size won’t help!

EBS-optimized

Queue Depth• Queue depth

An I/O

Amazon EBS

After it’s gone, it’s gone

AmazonEC2

AmazonEC2

Block SizeA bigger I/O

Amazon EBS

• > 16K

>16 KB? = more than one I/O

Block Size

A smaller I/O

AmazonEBS

• < 16K

< 16 KB? = no “bonus” random I/Os

AmazonEC2

Architecture – What Else Uses Network?

Workload/ Software

Typical block size

Random/Seq?

Typical correct EBS type

Max EBS @ 500 MB/s

Max EBS @ 1 GB/s

Max EBS @ CC/H

Oracle DB Configurable:2 KB –16 KBDefault 8 KB

random PIOPS RAID 0, 2 × 4000 PIOPS:~7800 IOPS

RAID 0, 4 × 4000 PIOPS:~15,600 IOPS

RAID 0, 24 × 4000 PIOPS:~96,000 IOPS

Microsoft SQL Server

8 KB w/ 64 KB extents

random PIOPS RAID 0, 2 × 4000 PIOPS:~7800 IOPS

RAID 0, 4 × 4000 PIOPS:~15,600 IOPS

RAID 0, 20 × 4000 PIOPS:~80,000 IOPS

MySQL 16 KB random PIOPS 1 x 4000 PIOPS:~4000 IOPS

RAID 0, 2 × 4000 PIOPS:~7,800 IOPS

RAID 0, 12 × 4000 PIOPS:~48,000 IOPS

PostgreSQL 8 KB random PIOPS RAID 0, 2 × 4000 PIOPS:~7,800 IOPS

RAID 0, 4 × 4000 PIOPS:~15,600 IOPS

RAID 0, 24 × 4000 PIOPS:~96,000 IOPS

MongoDB 4 KB serialized PIOPS RAID 0, 4 × 4000 PIOPS:~15,600 IOPS

RAID 0, 8 × 4000 PIOPS:~31,000 IOPS

RAID 0, 24 × 4000 PIOPS:~96,000 IOPS

Cassandra 4 KB random PIOPS RAID 0, 4 × 4000 PIOPS:~15,600 IOPS

RAID 0, 8 × 4000 PIOPS:~31,000 IOPS

RAID 0, 24 × 4000 PIOPS:~96,000 IOPS

GlusterFS 128 KB sequential PIOPS 1 × 4000 PIOPS:~500 IOPS

RAID 0, 2 × 4000 PIOPS:~1000 IOPS

RAID 0, 12 × 4000 PIOPS:~6,000 IOPS(768MB/s)

Cheat Sheet Sample! Storage Workloads on AWS

Smaller I/O (4 KB, 16 KB)

• Why are 4 KB I/O size in sequential operations driving greater than 4000 IOPS?

• Why is m1.large and m3.xlarge IOPS at 16 KB less than 4000 IOPS?

• Database needs 5000 ops/second. How many IOPS do I need to provision?

• What happens when customers want to burst beyond provisioned IOPS?

IOPS and BW performance at

QD 8

m1.large   M3.xlarge   m3.2xlarge

AvgIOPsAvg BW ( KB)   AvgIOPs

Avg BW ( KB)   AvgIOPs

Avg BW ( KB)

Write sequential                

4K 4146 16,587   5997 23,990   7767 31,068

16K 3712 59,402   4157 55,461   4153 60,332

Write random                

4K 4082 16,329   4433 17,733   4178 16,712

16K 3713 59,422   3743 53,813   4153 60,332

                 

Read sequential                

4K 5301 21,205   9232 36,929   13450 53,802

16K 3533 56,535   4796 56,824   4153 60,332

Read random                

4K 4538 18,154   5864 23,457   4177 16,711

16K 3510 56,168   3583 51,246   4153 60,332

Provisioned IOPS volumes process read/write in I/O sizes up to 16 KB and throttles on the number of IOPS provisioned

Results for 400 GB volume with 4000 IOPS at QD 8; EBS-optimized instances

Larger I/O (128 KB, 512 KB)• Why am I seeing only 462

IOPS on a volume?

• Why there is no difference in performance for random and sequential workloads?

• How should I configure 500 MB/s read or write throughput using PIOPS volumes

IOPS and BW performance at QD 8

m1.large   M3.xlarge   m3.2xlarge

AvgIOPs Avg BW(KB)   AvgIOPs Avg BW(KB)   AvgIOPs Avg BW(KB)

Write sequential                

128K 462 59,268   462 59,145   522 66,843

512K 115 59,292   115 59,278   130 66,804

Write random                

128K 462 59,265   462 59,241   522 66,843

512K 115 59,291   115 59,272   130 66,843

Read sequential                

128K 455 58,240   454 58,225   522 66,843

512K 113 58,003   114 58,589   130 66,843

Read random                

128K 455 58,236   454 58,215   522 66,843

512K 113 57,960   114 58,496   130 66,805

4000, 16 KB read/write per second, or 2000 32 KB read/write per second, or 1000 64 KB read/write per second…

Results for 400 GB volume with 4000 IOPS at QD 8

What about RAID?

Amazon EBS

Amazon EC2

• Crucial tool

• Don’t mix types

• No RAID 5, 6

• Use mdadm, LVM above that

=Amazon EBS

Amazon EC2

Architecting for Performance: RAID • Customers stripe number of

volumes to drive higher IOPS and throughput

– RAID 0 or RAID 10

• How should customers think about taking snapshots on a striped volume?

– Quiesce file systems and take snapshot

– Unmount file system and take snapshot

– Use OS-specific tools

IO Pattern Block Sizes Thread Write IOPS Write BW (MB) Read IOPS Read BW (MB)

Sequential 4K 8 33,500 134 48,250 19316K 8 13,875 222 48,063 7691M 1 247 247 823 823

 

Random4K 8 35,250 141 48,250 19316K 8 13,875 222 42,125 6741M 1 496 496 795 795

12×400 GB PIOPS volumes, pre-warmed, RAID 0 LVM, Stripe size 64 KB, attached to CR1 instance

• Leverage SSD instance type (hi1.4xlarge)o 2 × 1 TB SSD storage (ephemeral

storage)o Perfect for replicas

• If replicas on SSD instance types, disable integrity features such as fsync and full_page_writes on those hosts to improve performance

Performance – Extra-large Production Scale

Sta

ble

Testing Random 4 KB Reads

EBS PIOPS+ SSD

Performance / Stability Tips• Ext4 or XFS (understand journal impact!)

• nobarrier, noatime, noexec, nodiratime

• Raise file descriptor limits

• Set read-aheads low

• AWS business-level support – Trusted Advisor

• Amazon CloudWatch metrics in general

• SNAPSHOT SNAPSHOT SNAPSHOT

Pre-warming Amazon EBS volumes

• Typically 5%, extreme worst case of 50% performance reduction in IOPS and latency when volumes are used without pre-warming

– Performance is as provisioned when all the chunks are accessed

• Recommendation if testing or you have spare setup time:– Write to every 4 MB block before using new volumes

• Linux: DD • Windows: NTFS Full format

– Takes roughly an hour to pre-warm 1 TB 4 KB PIOPS volume– Be warned, can take up to a day for a 1 TB standard Amazon EBS volume

Architecting for Performance: Latency

• Performance requirements may be driven by IOPS or latency or both

• Recommendation is to start with queue depth of 4 and tune based on IOPS and latency requirement – Some customers may need lowest possible latency; this can be achieved at

queue depth of 1 or 2

• Very high queue depths ( >24) may decrease IOPS count as well as increase latency

Write Latency• Database applications care

about latency as much as IOPS delivered

• There is an interdependency among IOPS, queue depth, and latency

• Current guidance is queue depth of 1 for every 200 IOPS, but if latency-bound and write-heavy, 1:500 – 1:1000 is better.

1 4 8 12 16 20 24 28 32

AvgIOPS ( Count) 845 4152 4152.5 4177 4152.375 4176.16666666668

4177 4176.83333333333

4150.75

AvgTP90 ( ms) 3.13 1.47 2.03 3.56 3.62 5.54 6.18 7.48 7.71

250

750

1250

1750

2250

2750

3250

3750

4250

1

2

3

4

5

6

7

8

9

845

4152

3.13

1.47

2.03

3.56 3.62

5.54

6.18

7.487.71

16 KB random WRITE- M3.2Xlarge EBS-optimized

Wri

te I

OP

S

Latency

QD

Read Latency

• Reads can take advantage of a deeper queue

• Current guidance is queue depth of 1 for every 250 IOPS

• EBS-optimized provides predictable latency

1 4 8 12 16 20 24 28 32

AvgIOPS ( Count)

1864 4153 4152.5 4177 4120 2800.333333333

35

1965 1213.166666666

67

1088.625

AvgTP90 ( ms) 0.68 1.46 2.15 3.43 3.88 5.18 91.14 93.18 93.70

250

750

1250

1750

2250

2750

3250

3750

4250

5

15

25

35

45

55

65

75

85

95

1864

41534120

1965

0.68 1.46 2.15 3.43 3.88 5.18

91.14 93.18 93.70

16 KB random READ - M3.2Xlarge EBS-optimized

Rea

d

IOP

S

Latency QD

What About Performance Cost?cc2.8xlarge

24 @ 4 KBPIOPS

VS.

hi1.4xlarge hi1.4xlarge

$11773 on-demand, $10589 effective 3 YR reserved

$4538 on-demand,$1539 effective 3 YR reserved

If >20 KB IOPS read, choose hi1

If 3 YR, and >8 KB IOPS, choose hi1 If >10 KB write IOPS, TEST, but probably choose PIOPS

On demand, If <20 KB read IOPS, choose PIOPS

What about Capacity Cost?cc2.8xlarge

48x 1TB EBS

VS.

hs1.8xlarge hs1.8xlarge

$7312 on-demand, $6128 effective 3 YR reserved

$6734 on-demand,$2408 effective 3 YR reserved

If >43TB, or > 800MB/s, choose hs1

If 3 year, and >18TB, choose hs1

Tools available for tuning

1. EC2 Instance: Network transfer rate (Mbps)

2. EBS Optimized: EC2 instance option (On/Off)

3. PIOPS: Provisioned I/O operations per second (#)

4. Queue Depth: The number of outstanding I/Os (#)

5. Block Size: The size of each I/O (KB)

6. Workload: Block size, read/write ratio, serialization

EBS-optimized

Oh, YEAH!!

AmazonEC2

A “boatload” of I/O

From 1000 to 98,000: What does that mean?

• What was advanced before is now easy

• What was “impossible in cloud” is now normal

• Amazon EBS increases two orders of magnitude – WOW!

So.. what storage workloads?• Relational DB

• NoSQL DB

• Data warehousing

• Files

• Media

© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Maximizing EC2 and Elastic Block Store Disk Performance

Todd Varland, Solutions Architect, AWS

Thank you!

1EBSRAID

AZFS

Volumes

PRIMARYController

Availability Zone 1SoftNAS Instance A

SECONDARYController

Availability Zone 2SoftNAS Instance B

EIP

NFS CIFS iSCSI

2 N

SnapReplicate

1EBSRAID

AZFS

Volumes

2 N… …

BlockReplication

SnapReplicate™

AWS Marketplace: Softnas.comSnapshots of multiple Amazon EBS volumes in a RAID array

HA Monitor

Q1 2014