red hat storage day la - performance and sizing software defined storage
TRANSCRIPT
PERFORMANCE AND SIZINGSOFTWARE DEFINED STORAGEKyle BaderRed Hat Storage Day, Los AngelesAugust 2016
$> whois kyle bader
Senior Solution ArchitectRed Hat
DATA CHALLENGES
Exponential growth in digital content increases pressure on capacity, scalability, and cost.
The need for access to data from anywhere, anytime, on any device requires unprecedented agility.
Modern services require the flexibility to store data on-premises or in the cloud.
Growing content requires advanced data protection that ensures integrity & high availability at very large scale.
1
2
3
4
Traditional StorageComplex proprietary silos
Open, Software-Defined StorageStandardized, unified, open platforms
Custom GUI
Proprietary Software
ProprietaryHardware
StandardComputersand DisksSt
anda
rdHa
rdwa
reOp
en S
ourc
eSo
ftwar
e
Ceph Gluster +++
Control Plane (API, GUI)
ADMIN USER
THE FUTURE OF STORAGE
ADMIN
USER
ADMIN
USER
ADMIN
USER
Custom GUI
Proprietary Software
ProprietaryHardware
Custom GUI
Proprietary Software
ProprietaryHardware
FLEXIBILITY IS CRUCIAL
Server-based storage uses software and standard hardware to provide services traditionally provided by single-purpose storage
appliances, providing increased agility and efficiency.
DISTRIBUTED CLUSTER OF SERVERS
MEDIA
MEDIA
MEDIA
MEDIA
MEDIA
MEDIA
MEDIA
APPLIANCE
MEDIA
MEDIA
APPLIANCE
MEDIA
MEDIA
APPLIANCE
MEDIA
MEDIA
USER USER USER
SERVER-BASED STORAGE
USER USER USER
STORAGE APPLIANCE
COMPUTE NODE + + + +
+
STORAGE NODE
COMPUTE NODE + + + +
+ + + +
+
+
+
+
+
+
+
+
VIRTUALIZED STORAGE SCALES BETTER
STANDARD SAN/NAS IS ON THE DECLINE
Server-based storage is “will account for over 60% of shipments long term.”
“By 2016, server-based storage solutions will lower storage hardware costs by 50% or
more.”Gartner: “IT Leaders Can Benefit From Disruptive Innovation in the Storage Industry”
Credit Suisse Storage Update, September 3, 2015
Changing workloads drive the need for flexible, economical server-based
storage.
WW DEPLOYED CAPACITY (TB)
2010
2011
2012
2013
2014
2015
(est)
2016
(est)
Source: IDC
0%
60%
20%
40%
80%
100%
Internal Capacity
External Capacity
STORAGE ORCHESTRATION
Storage orchestration is the ability to provision, grow, shrink, and decommission storage resources on-demand and programmatically,
providing increased control and integration of storage into a software-defined data center.
WEB CONSOLEA browser interface designed for managing distributed
storage
APIA full API for automation and integration with outside
systems
COMMAND LINEA robust, scriptable command-
line interface for expert operators
PROVISION INSTALL CONFIGUR
E TUNE MONITOR
Full lifecycle management for distributed, software-defined data services
A RISING TIDE
“By 2020, between 70-80% of unstructured data will be held on lower-cost storage managed by SDS”
“By 2019, 70% of existing storage array productswill also be available as software only versions”
Innovation Insight: Separating Hype From Hope for Software-Defined Storage
Innovation Insight: Separating Hype From Hope for Software-Defined Storage 2013
2014
2015
2016
2017
2018
2019
$1,349M
$1,195M
$1,029M
$859M
$706M
$592M
SDS-P MARKET SIZE BY SEGMENT
$457M
Block StorageFile StorageObject StorageHyperconverged
Source: IDC
Software-Defined Storage is leading a shift in the global storage
industry, with far-reaching effects.
THE BALANCE
InflexibleExpensive at large scale
DurableConvenient
FlexibleEconomical at large scale
DurablePowerful
Appliances are suitable for small-scale, workloads, but they do not
scale economically.
Software-defined storage has a learning curve, but bring
performance and economy at petabyte scale.
THE ROBUSTNESS OF SOFTWARE
Software is more flexible than hardwareSoftware can do things hardware appliances can’t. SDS brings the flexibility of software to the enterprise storage world.• Can be deployed on bare metal, inside containers, inside
VMs, or in the public cloud.• Can deploy on a single server, or thousands, and can be
upgraded and reconfigured on the fly.• Grows and shrinks programmatically to meet changing
demands
BUILDING ON PROVEN HARDWARE
Hardware is hard, and we got you coveredTested software defined storage solutions, for repeatable success.• Ceph Hardware Configuration Guide• Ceph Hardware Selection Guide• Ceph Performance and Sizing Guide - Supermicro• Ceph Performance and Sizing Guide - Quanta QCT
OPTIMIZATION CRITERIA
IOPS Optimized
ThroughputOptimized
CapacityOptimized
• Lowest cost per IO• Highest IOPS• Meet minimum fault domain
requirement
• Lowest cost per unit of throughput• Highest throughput• Highest throughput per watt/BTU• Meet minimum fault domain
requirement
• Lowest cost per TB• Lowest watt/BTU per TB• Meet minimum fault domain
requirement
• Typically block storage• Replication• MySQL for OpenStack tenants
• Block and object storage• Replication or erasure coded• Active performance storage for
video, audio, and images• Streaming media
• Typically object storage• Erasure coding dominant• Media archives• Data lake
• Shared, elastic storage pool• Dynamic DB placement• Flexible volume resizing• Live instance migration• Backup to object pool• Read replicas via copy-on-write
snapshots
MySQL ON CEPH STORAGE CLOUDOPS EFFICIENCY
MYSQL-ON-CEPH PRIVATE CLOUDFIDELITY TO A MYSQL-ON-AWS EXPERIENCE
• Hybrid cloud requires public/private cloud commonalities
• Developers want DevOps consistency• Elastic block storage, Ceph RBD vs. AWS EBS• Elastic object storage, Ceph RGW vs. AWS S3• Users want deterministic performance
HEAD-TO-HEADPERFORMANCE
30 IOPS/GB: AWS EBS P-IOPS TARGET
HEAD-TO-HEAD LABTEST ENVIRONMENTS
• EC2 r3.2xlarge and m4.4xlarge• EBS Provisioned IOPS and GPSSD• Percona Server
• Supermicro servers• Red Hat Ceph Storage RBD• Percona Server
OSD Storage Server Systems5x SuperStorage SSG-6028R-OSDXXX
Dual Intel Xeon E5-2650v3 (10x core)32GB SDRAM DDR32x 80GB boot drives 4x 800GB Intel DC P3700 (hot-swap U.2 NVMe)1x dual port 10GbE network adaptors AOC-STGN-i2S 8x Seagate 6TB 7200 RPM SAS (unused in this lab)Mellanox 40GbE network adaptor(unused in this lab)
MySQL Client Systems12x Super Server 2UTwin2 nodes
Dual Intel Xeon E5-2670v2 (cpuset limited to 8 or 16 vCPUs)64GB SDRAM DDR3
Storage Server Software:Red Hat Ceph Storage 1.3.2Red Hat Enterprise Linux 7.2Percona Server
5x OSD Nodes 12x Client Nodes
Shared 10G SFP+ Networking
Monitor Nodes
SUPERMICRO CEPHLAB ENVIRONMENT
P-IOPS _x000d_m4.4xl P-IOPS_x000d_ r3.2xl GP-SSD _x000d_r3.2xl0
1000
2000
3000
4000
5000
6000
7000
8000
9000
7996 7956
950
1680 1687
267
100% Read100% Write
SYSBENCH BASELINE ON AWS EC2 + EBS
01000020000300004000050000600007000080000
7996
67144
40031
1680 5677 1258
20053
4752
100% Read
100% write
70/30 RW
SYSBENCH REQUESTS PER MYSQL INSTANCE
CONVERTING SYSBENCH REQUESTS TO IOPS READ PATH
X% FROM INNODB BUFFER POOL
IOPS = (READ REQUESTS – X%)
SYSBENCH READ
CONVERTING SYSBENCH REQUESTS TO IOPS WRITE PATH
SYSBENCH WRITE
1X READ
X% FROM INNODB BUFFER POOL
IOPS = (READ REQ – X%)
LOG, DOUBLE WRITE BUFFER
IOPS = (WRITE REQ * 2.3)
1X WRITE
P-IOPS _x000d_m4.4xl P-IOPS _x000d_r3.2xl GP-SSD _x000d_r3.2xl0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
30.0 29.8
3.6
25.6 25.7
4.1
100% Read100% Write
AWS IOPS/GB BASELINE: ~ AS ADVERTISED!
IOPS/GB PER MYSQL INSTANCE
0
50
100
150
200
250
300
30
252
150
26
78
19
MySQL IOPS/GB ReadsMySQL IOPS/GB Writes
FOCUSING ON WRITE IOPS/GBAWS THROTTLE WATERMARK FOR DETERMINISTIC PERFORMANCE
0102030405060708090
26
78
19
A NOTE ON WRITE AMPLIFICATIONMYSQL ON CEPH – WRITE PATH
INNODB DOUBLEWRITE BUFFER
CEPH REPLICATION
OSD JOURNALING
MYSQL INSERT
X2
X2
X2
EFFECT OF CEPH CLUSTER LOADING ON IOPS/GB
Ceph c
luster
_x00
0d_(1
4% ca
pacit
y)
Ceph c
luster
_x00
0d_(3
6% ca
pacit
y)
Ceph c
luster
_x00
0d_(7
2% ca
pacit
y)
Ceph c
luster
_x00
0d_(8
7% ca
pacit
y)0
20406080
100120140160
78
3725 19
134
72
37 36
100% Write70/30 RW
IOPS
/GB
05
10152025303540
18 18 19
6
34 34 36
8
100% Write70/30 RW
IOPS
/GB
CONSIDERING CORE-TO-FLASH RATIO
HEAD-TO-HEADPERFORMANCE
30 IOPS/GB: AWS EBS P-IOPS TARGET
25 IOPS/GB: CEPH 72% CLUSTER CAPACITY (WRITES)78 IOPS/GB: CEPH 14% CLUSTER CAPACITY (WRITES)
HEAD-TO-HEADPRICE/PERFORMANCE
$2.50: TARGET AWS EBS P-IOPS STORAGE PER IOP
IOPS/GB ON VARIOUS CONFIGS
-
10
20
30
40
50
60
70
80
90
31
18 18
78
AWS EBS Provisioned-IOPSCeph on Supermicro FatTwin 72% CapacityCeph on Supermicro MicroCloud 87% CapacityCeph on Supermicro MicroCloud 14% Capacity
IOPS
/GB
(Sys
benc
h W
rite)
$/STORAGE-IOP ON THE SAME CONFIGS
$-
$0.50
$1.00
$1.50
$2.00
$2.50
$3.00
$2.40
$0.80 $0.78 $1.06
AWS EBS Provisioned-IOPSCeph on Supermicro FatTwin 72% CapacityCeph on Supermicro MicroCloud 87% CapacityCeph on Supermicro MicroCloud 14% Capacity
Stor
age
$/IO
P (S
ysbe
nch
Writ
e)
HEAD-TO-HEADPRICE/PERFORMANCE
$2.50: TARGET AWS P-IOPS $/IOP (EBS ONLY)$0.78: CEPH ON SUPERMICRO MICRO CLOUD CLUSTER
8x Nodes in 3U chassisModel: SYS-5038MR-OSDXXXP
Per Node Configuration:CPU: Single Intel Xeon E5-2630 v4Memory: 32GB NVMe Storage: Single 800GB Intel P3700 Networking: 1x dual-port 10G SFP+
+ +
1x CPU + 1x NVMe + 1x SFP
SUPERMICRO MICRO CLOUDCEPH MYSQL PERFORMANCE SKU