dmx performance

1© Copyright 2008 Interscape Technologies Inc. All rights reserved.

Alok Jain

Performance Aspects of Open Systems on

EMC Platforms (Symmetrix)

09/12/08


Agenda

�Storage Area Networks from Performance Perspective

�Open Systems and Performance

�EMC Storage Platforms – Symmetrix

�Open Systems Tools

�Performance Metrics for Open Systems

�Performance Metrics for EMC Storage

�Performance Analysis using tools

�Storage Consolidation Planning & Considerations

�Use Case Study (time permitting)


Storage Performance (SNIA)

�Most data centers are dealing with ever increasing amounts of data storage. Although the need for storage seems insatiable, array performance typically plateaus or worse, degrades post installation. Storage performance tuning is a necessary and ongoing activity…..


Storage Area Networks and Performance

Storage IO performance issues can occur at any of these layers.

Data replication to remote site


Open Systems and Storage Performance

HARDWAREHARDWARE

DRIVER

LVMLVM

File System

SYMMETRIX

Path Manager

Host

Application

Symmetrix sees something very different than the initial size of the I/O!

Hardware delivers the I/O as requested

Driver may break up I/O (max_phys and other parameters)

Path manager may affect I/O channel and I/O size

File System changes I/O to conform to File System block size, fragment size, or memory page size

Database changes I/O subject to db_block_size, multiblock_read_count, and other parameters. Database

Application performs an I/O

SYMMETRIX

LVM breaks up according to stripe width & I/O parameter settings


Application and System Performance – Key Metrics

�Three Ways to Measure Application Performance

� Throughput (MB/s)

� Number of transactions per second

� Application run time

�Four Ways to Measure IO Sub-System Performance� IO/sec (IO operations per sec)

� MB/sec (Bytes transferred/sec)

� I/O Service Times (average time to do an I/O)

� I/O Queue Lengths (outstanding IO waiting to be sent)


Types of IO & Performance Impacting Points

�There is a limited number of basic I/O operations� Random read

� Sequential read

� Random write

� Sequential write

�Each type has “cache hit” and “cache miss” variants

�Block Size impacts the throughput (IO/sec) and Bandwidth (MB/s)


Application Work Load Characteristics

�All applications perform the following:� Read/Write data to/from disk systems (milliseconds)

� Process data in CPU (microseconds)

� Write data back to disk system (milliseconds)

Performing an IO is an order of magnitude time consuming compared to a CPU operation

�You need to know� Overall I/O rate

� % Read + read size

� % Write + write size

� % Random, % sequential

� % Cache hit (locality of reference)


Physical Drives – Key Consideration

� Number of disks and speed of disks can limit IO Subsystem performance

� The 7.2K RPM drives are good for sequential and large I/O sizes. A little slower at random access than the 10K RPM drives

� The 10K RPM drives give good, all-around performance

� 15K RPM Drives boost random I/O access rates� 30% more random-access, small transfers than the 10K RPM

drives

� They do not benefit much in large (64KB +) or sequential operations, as transfer rates are not as affected by rotational speed

� If I/O is hitting the cache, they’re all the same speed� For writes, disk type is unimportant as long as write cache is not

saturated

� Prefetch works at about the same rate for all of them

� What differs is how well they do random reads, and how fast theyflush random writes from the cache


Drives can be roughly divided into “families”

�The major distinction is the rotational speed� Faster rotation generally also means faster seek

Not Fast

146GB 15k

RPM

73GB 15k

RPM

300GB 10k

RPM

146GB 10k

RPM

73GB 10k

RPM

500GB 7200

RPM

Drive Capacity (Gbyte) 146 73 300 146 73 500

Internal data rate (MB/sec) 68 - 114 68 - 114 47 - 94 47 - 94 47 - 94 47 - 94

Disc Rotational Speed (r/min) 15,000 15,000 10,000 10,000 10,000 7,200

Avg Rotational Latency (ms) 2.00 2.00 3.00 3.00 3.00 4.17

Avg Read Seek (ms) 3.50 3.50 4.70 4.70 4.70 8.50

Avg Write Seek (ms) 4.00 4.00 5.40 5.40 5.40 9.50

Read Single Track Seek (ms) 0.27 0.27 0.20 0.20 0.20 0.80

Write Single Track Seek (ms) 0.45 0.45 0.50 0.50 0.50 1.00

Read Full Stroke Seek (ms) 7.40 7.40 9.70 9.70 9.70 16.00

Write Full Stroke Seek (ms) 7.90 7.90 10.40 10.40 10.40 16.55

Fastest Fast

Specification


RAID Level Considerations for IO Sub-Systems

High data availability. High cost. Best performance for random reads & writes

Striped + MirroredRAID 1+0

Highest data availability. Low cost. Slower than R5 performance

Striped set with dual distributed parity

RAID 6

High data availability. Lower cost. Cost effective performance

Striped set with distributed parity

RAID 5

Higher data availability. High cost. Best Sequential IO performance

Mirrored setRAID 1

Lowest data availability. Lowest cost. Highest performance. Not Supported in EMC Symmetrix

Striped set without parityRAID 0

CommentsRAID Group Configuration

RAID Level


RAID 1

� Writes go to two drives

� Read from 1 drive� DMSP policy

� High level of protection

� Most expensive $$$� 2x raw storage

Channel

ChannelDirector

Directory

�Global Memory

DiskDirector

Disk

Disk

M1

M2

EMC Symmetrix – RAID 1 (Sample Symm View)


Striping – At Host Level & Array level (Plaids for Best Performance)

�LUN = Logical Unit Number (drive letter/#)

�LUNs striped across multiple RAID groups (of same type)� Eliminate hot RAID groups, hot drives, hot back-end

links

� Variations include, RAID 0+1, 1+0, 5+0, 10, or 50


Open Systems Performance Measurement Tools

GUI based (with CLI options also)

PerfmonWindows

Iostat –x 5

Sar –d (many options)Iostat & sarAIX

Iostat –xt 5

Sar –d (many options)Iostat & sarLinux

Iostat –xtczn –T d 5

Sar –d (many options)Iostat & sarSolaris

Command ExampleToolOS Platform


Iostat – Unix Based Systems (Solaris)

ThroughputReads/sec & Writes

Cmds in Wait queue

Cmds activelyBeing executed

Wait tiimeIn queue

Actual ServiceTime

BandwidthRead & Write KB/sec

From Performance perspective the bottlenecks can be identified by looking at Wait Queues, Wait Service Time and

Active Service Time


Iostat – Unix Based Systems (Solaris)

�Performance Throughput Indicators� Reads/sec & Writes/sec

� Read Kbytes/sec & Write Kbytes/sec

�Performance Bottlenecks Indicators� Active Service Time – This can be application

dependent. It should be compared to a baseline to determine the changes when the problem occurs

� Wait Queues present – signifies IO sub-system is not able to keep up with demand at the LUN level

� Wait Service Time – Any wait service time will introduce IO latency


Sar – Unix based Systems (Solaris)

AvgQueues

Throughput Total IO/sec

Bandwidth total blocks/sec+

Avg. WaitTime (ms)

Avg. ServiceTime (ms)

SAR –d output. The metrics are different compared to IOSTAT output. Reads and writes data is shown together

%busy indicator is indicative only.


Perfmon – Windows Based Systems

Windows based systems can use Perfmon to do performance analysis


Windows Host – EMC Symmetrix Array

Windows Host Drives mapped to EMC Symmetrix physical disks. The color shows how utilized the physical disk spindles are in terms of IO performance.

This provides a comprehensive view of Host devices in the Storage Array


EMC Storage Platforms Performance Measurement Tools

GUI based

Navicli –h ip_address …..

Navisphere Analyzer

Navicli (CLI)Clariion

WLA is GUI based

Symstat –sid 1234 –c 5 –I 5

Work Load Analyzer (WLA)

Symstat (CLI)Symmetrix

ExampleToolEMC Platform


Symmetrix Components from Performance Perspective

� Symmetrix DMX Critical Component� %Cache Hits

� Fiber Adapter Channels

� Disk Directors

� Remote Fiber Director

� Device Utilizations

� Disk Utilizations (physical)

� Symmetrix Key Performance Metrics (at eac component level)� Read/write IO/sec

� Read/write Kbytes/sec

� %cache hits

� Block Sizes

� Write Pending Slot Counts


EMC Symmetrix Global Memory Management & Data Flow


EMC DMX-4 Symmetrix Array Architecture

Cache

Front-EndTo Hosts

Back-EndTo Disks


EMC Symmetrix – Tiering within arrays


EMC DMX-4 Symmetrix Disk Quadrant Layout

All drive types in all quadrants to leverage all Disk Adapters for Tiering


EMC Symmetrix – Performance Viewer (Work Load Analyzer)


EMC Symmetrix IO/sec & MB/sec Metrics

Monitor overall IO/sec and MB/sec for the whole array. Confirm the storage array is not over loaded

Reads MB/sec, Write MB/sec, Total MB/sec

Read IO/sec, Write IO/sec, Total IO/sec


EMC Symmetrix Write Pending Slots Vs. Max WP Slots

Monitor overall Write Pending slot counts for the whole array. Crossing Max thresholds will impact performance


EMC Symmetrix %Cache Hits

Monitor %Cache Hits. For Open Systems anything below 70% may impact performance negatively


EMC Symmetrix – Front-End Fiber Adapter CPU IOPS

Monitor Fibre Adapter CPU IO/sec on each individual FA CPUIdentify busiest FA CPUs


EMC Symmetrix – Front-End Fibre Adapter FA Port Utilization (based on IO/sec)

Identify FA Ports that are highly utilized. Anything over 90% may be of concern


EMC Symmetrix – Remote Adapters IOPS & MBPS (used for remote replication)

Identify RA Ports that are doing heavy IOs. Generally most RA’s will have similar IO profiles


EMC Symmetrix – Disk Adapters (DA)

Identify DA Ports that are doing heavy IOs. Generally most DA’s will have similar IO profiles if the hardware configuration is setup properly.


EMC Symmetrix – Devices – Sampled Average read and write times

Identify devices that are doing very high reads and writes. These devices then need to be mapped to the hosts that they are provisioned to


EMC Symmetrix – Disk SCSI Command Loads

Identify physical disks that are very busy. Any 10K RPM disk doing over 120-140 IOPS points to hot disks which will create bottlenecks


EMC Symmetrix – Disk to Device Mappings (Identify Bad Neighbors)

Identify busiest devices on the actual physical disks. This will point to hosts that may be starving other hosts for IO


EMC Symmetrix – Component %Utilization HEAT Chart

This provides a one page view into the Symmetrix array.The %utilization colors points to how hot each of the components are


Questions?


�Contact details

U.S. Office:200 Centennial Avenue,Suite 200, Piscataway, NJ 08854Office: 1.732.868.6320Fax: 1.973.302.8135

Alok JainCell: 848.248.0100Office: 732.868.6320Email id.: [email protected]


Storage Consolidation Analysis

Considerations & Use Case


IT Assets in General are Poorly Used

Typical Utilization Rates

Servers: 5-15%

PCs: 10-20%

Direct-attach storage: 20-40%

Network storage: 60-80%

A typical x86 serverconsumes between 30% and 40%of its maximum power when idle

Sources: VMware, Microsoft, EMC


Storage Consolidation Energy Benefits

�Storage consolidation saves energy

�Fewer and larger storage arrays� Consolidate and optimize

workloads

� Maximize storage utilization

�Requires understanding and planning� What are workloads?

� What are storage array capabilities?

� What are the performance characteristics?

� What are the data protection and business continuity requirements?

42


RAID 3+1 Annual Energy Cost Usable Terabyte vs

Drive Size

0.00

200,000.00

400,000.00

600,000.00

800,000.00

1,000,000.00

0 500 1000

Usable Terabytes (TB)

En

erg

y C

ost

@

$0.1

5/k

Wh

r

73 GB Yearly EnergyCost 3+1

146 GB YearlyEnergy Cost 3+1



Higher density disks brings the TCO down


Use Case: Storage Savings on Capital, Mgmt and Energy

Customer Example: Major Wireless Provider

1,296,480 kWh - Annual Energy Savings - $194,472

44 Mid-tier systems consolidated to 2 High-end systems

$222,066Annual

Cost @.15 kWh

$416,538

169 kWhPower and

Cooling317 kWh

High-end Array934 TB in 146 GB and 300 GB

drives

Mid-Tier885 TB in 146 GB drives


Use Case: Driving Factors Migration & Consolidation Planning

�Client is migrating Data Center form NY to NJ for various business reason

�Client would like to perform a detailed storage consolidation analysis as part of decision on migration strategy

�Client wants to look at their Tier-1 Storage environment which is mostly on EMC Symmetrix with SRDF Replication to another Data Center

�The consolidation analysis needs to model� Overall Storage Array Performance� Storage Capacity Consolidation Requirements� Technical Architecture at component levels

� LUN Limitations� Host Fan-Out ratios� Device masking thresholds


ALL EMC Symmetrix Performance – System IOPS & MBPS

S. No

Symmetrix

ID

Symmetrix

Pair Alias Location

Symm

Mode

Max

(IO/Sec)

95th

Percentile

(IO/Sec) Avg (IO/Sec)

Max

(MB/Sec)

95th

Percentile

(MB/Sec)

Avg

(MB/Sec) TimeStamp

1 190100031 190100032 NA DC 1 R1 35474 26244 16614 531 408 230 7/11/2008 21:452 190100011 190100012 NA DC 1 R1 32126 23554 12358 726 483 217 7/13/2008 12:45

3 190100015 190100016 NA DC 1 R1 30582 21398 12666 45 347 176 7/11/2008 22:154 187750019 187750020 NA DC 1 R1 24750 21812 12715 316 227 128 7/10/2008 22:155 187720023 187720024 NA DC 1 R1 16707 10049 5685 161 97 43 7/13/2008 0:45

6 187720025 187700026 NA DC 1 R1 14022 9341 4005 210 118 46 7/11/2008 22:007 187720027 187720028 NA DC 1 R1 13298 8976 4683 195 150 68 7/10/2008 21:15

8 187720013 187700014 NA DC 1 R1 12849 8696 4735 220 114 53 7/10/2008 21:309 287750029 287750030 NA DC 1 R1 12215 7796 4065 193 140 51 7/11/2008 22:30

10 187720021 187720022 NA DC 1 R1 11512 8739 4308 215 125 50 7/11/2008 21:30

11 187720017 187720018 NA DC 1 R1 9977 7741 4453 145 101 48 7/12/2008 2:3012 190100032 NA DC 2 R2 17003 11739 7147 249 168 78 7/11/2008 22:4513 190100016 NA DC 2 R2 14983 11123 6784 173 97 41 7/12/2008 19:30

14 190100012 NA DC 2 R2 11703 7677 3672 295 207 154 7/11/2008 23:1515 187750020 NA DC 2 R2 10755 6745 4174 90 70 28 7/12/2008 22:30

16 187720024 NA DC 2 R2 7720 3197 766 99 47 10 7/11/2008 20:1517 187720022 NA DC 2 R2 5764 2858 1381 27 18 8 7/12/2008 22:4518 287750030 NA DC 2 R2 5177 3356 1634 43 22 11 7/10/2008 22:30

19 187720018 NA DC 2 R2 5048 2180 971 24 18 6 7/10/2008 21:4520 187700014 NA DC 2 R2 2859 544 188 38 7 2 7/12/2008 19:4521 187720028 NA DC 2 R2 2387 1360 308 10 5 1 7/10/2008 20:15

22 187700026 NA DC 2 R2 212 67 46 3 0 0 7/11/2008 21:15


ALL EMC Symmetrix Performance – Max System IOPS & MBPS – R1+R2 View

Aggregate IOPS & MBPS for each interval across the pairs. This reflects a more realistic failed over view

14811510R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0017-0018 11

21412927R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0021-0022 10

15413213R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0013-0014 9

21014066R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0025-0026 8

16514388R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0029-0030 7

9614621R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0027-0028 6

15117205R1-R2 DC1 - DC2 2000S-M2 - 2000S-M2 0023-0024 5

32930579R1-R2 DC1 - DC2 2000P-M2 - 2000P-M2 0019-0020 4

88935214R1-R2 DC1 - DC2 DMX3-24 - DMX3-24 0011-0012 3

45939648R1-R2 DC1 - DC2 DMX3-24 - DMX3-24 0015-0016 2

72043278R1-R2 DC1 - DC2 DMX3-24 - DMX3-24 0031-0032 1

System Max Total MBPS (R1+R2)

System Max Total IOPS (R1+R2) Symm Mode Location Model Source Array No


Consolidation Analysis – Target Requirements

4+4RAID-1+010K400GBDMX-4T3



3+1RAID-515K300GBDMX-4T1

Target

RAID

Config

Target

RAID

Level

Target

Disk RPM

Target

Disk Size

Target

Model

Target

Config

ID


Symmetrix Performance IOPS & MBPS Profiles


Source Storage Arrays For Consolidation Targets


Target Storage Array Specifications


Source Storage #1 Performance Baseline


Source Storage #2 Performance Baseline


Symmetrix Sample – Aggregated System IOPS & MBPS

Sample shows two EMC Symmetrix IOPS and MBPS Consolidated View


Symmetrix Sample – Aggregated RA IOPS & MBPS

Sample shows two EMC Symmetrix Replication Channel Consolidated View


Symmetrix Sample – FA IOPS, MBPS, Masked Devices & FAN-OUT Analysis

Sample shown for two source array consolidation analysis view


Target Storage Array – Performance Heat Chart (Model)


Symmetrix Sample – Consolidated Target Storage Requirements

Sample shown for 2 source consolidation analysis view. Shows types of total device configs needed


ALL Symmetrix –2 to 1 Consolidation with RAID 1+0 – Overall Status

OK72%48030072043278Bi-DirDC1-DC2 DMX3-24 -DMX3-24 0031-0032 6

16514388Bi-DirDC1-DC22000S-M2 -2000S-M2 0029-0030 OK24%480300

9614621Bi-DirDC1-DC2 2000S-M2 -2000S-M2 0027-0028

5

21014066Bi-DirDC1-DC2 2000S-M2 -2000S-M2 0025-0026 OK27%480300

15117205Bi-DirDC1-DC22000S-M2 -2000S-M2 0023-0024

4

21412927Bi-DirDC1-DC2 2000S-M2 -2000S-M2 0021-0022 OK40%480300

32930579Bi-DirDC1-DC22000P-M2 -2000P-M2 0019-0020

3


45939648Bi-DirDC1-DC2 DMX3-24 -DMX3-24 0015-0016

2


88935214Bi-DirDC1-DC2 DMX3-24 -DMX3-24 0011-0012

1

Status

Target Storage

%Utilized

Recomm. Disk Count

Target Disk Size (GB)

System Max Total MBPS

(R1+R2)

System Max Total IOPS

(R1+R2) Symm Mode Location Model

Source Array Pair

Consol Group


Consolidation End Results

� 22 EMC DMX-2000 Symmetrix � 10 EMC DMX-4 Symmetrix

� 73GB 15K x 6336 Drives � 300GB 10K x 4800 Drives

� 90% Storage utilized � ~35% Storage Utilized with plenty of growth available

� ~500 Hosts � ~500 Hosts

� Data Migration using Symmetrix SRDF Array based utility

� Lowered Operational and Management Cost

� Much less spinning disks with plenty of growth left

� Performance modeling helped in minimizing performance impact


Questions?

dmx performance

Documents