ibm virtualization engine ts7720 and ts7740 … virtualization engine ts7700 release 3.1 performance...

© Copyright IBM Corporation

IBM® Virtualization Engine™ TS7720 and TS7740 Release 3.1 Performance White Paper Version 2.0

By Khanh Ly and Luis Fernando Lopez Gonzalez Tape Performance IBM Tucson

IBM System Storage™ March 15, 2014

IBM Virtualization Engine TS7700 Release 3.1 Performance White Paper Version 2.0

© Copyright 2014 IBM Corporation

Table of Contents

Table of Contents ..............................................................................................................2

Introduction.......................................................................................................................3

TS7700 Performance Evolution .......................................................................................4

TS7700 Copy Performance Evolution .............................................................................6

Hardware Configuration..................................................................................................8

TS7700 Performance Overview .......................................................................................9

TS7700 Basic Performance.............................................................................................12

Additional Performance Metrics ...................................................................................25

Performance at Distance.................................................................................................29

Performance Tools ..........................................................................................................34

Conclusions......................................................................................................................35

Acknowledgements..........................................................................................................36



Introduction

This paper provides performance information for the IBM Virtualization Engine™ TS7720 and TS7740, which are the two current products in the TS7700 Virtualization Engine family of products. This paper is intended for use by IBM field personnel and their customers in designing virtual tape solutions for their applications. This is an update to the previous TS7700 paper dated March 27, 2013 and reflects changes for release 3.1, which introduces 8Gb FICON with an additional 16GB memory expansion (total 32GB memory), and the doubling of supported FICON connections from 4 to 8. The 8Gb FICON improves TS7700 performance significantly, over 120% in some cases.

This paper is published under two different R 3.1 minor versions. The two versions are outlined below:

� Version 1: � Standalone TS7720 VEB/CS9 with different drawer counts (1, 2, 3, 6,

10, 26, and 42 drawers) � Standalone TS7740 V07/CC9 with 1 and 3 drawer counts � Grid configurations with 4x1Gb links. � Grid configuration with 2x10Gb links � Grid configuration with 2x10Gb links with single TS7720 expansion

frame � Copy performance over distance using two 10Gb links.

� Version 2: � Standalone TS7720 VEB/CS9/10 drawers/8x8Gb FICON with

workload driven from a z10 host only supporting 4Gb FICON � Standalone TS7740 V07/CC9/2 drawers � Performance vs. virtual volume sizes � Copy performance over distance using four 1Gb links.

The following are performance related changes in release R 3.1:

� 8Gb FICON Support � 8 channels (dual ports per card) � 4 channels (single port per card) � 32GB memory

� TS7720 Additional Expansion Frame Support � 1PB TS7720 (2

nd expansion frame)

Product Release Enhancements



TS7700 Performance Evolution

The TS7700 architecture continues to provide a base for product growth in both performance and functionality. Figures 1 and 2 shows the write and read performance improvement histories.

Figure 1. VTS/TS7700 Standalone Maximum Host Write Throughput. All runs were

made with 128 concurrent jobs, each job writing 800 MiB (300 MiB volumes @ 2.66:1

compression) using 32KiB blocks, and QSAM BUFNO = 20.

Notes:

• nDRs : number of cache drawers

• m x m Gb:

o 4x4Gb -- four 4Gb FICON channels

o 8x8Gb – eight 8Gb FICON channels (dual ports per card)

o 4x1x8Gb – four 8Gb FICON channels (single port per card)

• CS9* -- the new higher-performance 3TB drive (see figure 5)

0 500 1000 1500 2000 2500

Host MB/s (uncompressed)

B10 VTS (4xFICON)

B20 VTS (8xFICON) W/FPA

TS7740 V06/CC6/CX6 R 1.3 (4DRs/4x2Gb/z900)



TS7720 VEA/CS7/XS7 R 1.5 (7DRs/4x4Gb/z10)







TS7720 VEB/CS8/XS7 R 2.0 (7DRs/4x4Gb/z10)




TS7720 VEB/CS9/XS9 R 3.0 (10-DRs/4x4Gb/z10)


TS7720 VEB/CS9*/XS9 R 3.1 (10-DRs/4x1x8Gb/z10)

TS7740 V07/CC9/CX9 R 3.1 (3DRs/4x1x8Gb/z10)

TS7720 VEB/CS9/XS9 R 3.1 (10-DRs/8x8Gb/z10)

TS7720 VEB/CS9*/XS9 R 3.1 (10-DRs/8x8Gb/z10)




Standalone VTS/TS7700 Write Performance History

Sustained Write Peak Write

Standalone Performance Evolution



Figure2. VTS/TS7700 Standalone Maximum Host Read Hit Throughput. All runs were made with 128 concurrent jobs, each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32KiB blocks, and QSAM BUFNO = 20. See definition for Read Hit in the section “TS7700 Performance Overview”.

Notes:

• nDRs : number of cache drawers

• m x m Gb:

o 4x4Gb -- four 4Gb FICON channels

o 8x8Gb – eight 8Gb FICON channels (dual ports per card)

o 4x1x8Gb – four 8Gb FICON channels (single port per card)


0 500 1000 1500 2000 2500

Host MB/s (uncompressed)

B10 VTS (4xFICON) B20 VTS (8xFICON) W/FPA

TS7740 V06/CC6/CX6 R 1.3 (4DRs/4x2Gb/z900)TS7740 V06/CC6/CX6 R 1.5 (4DRs/4x2Gb/z900)TS7740 V06/CC6/CX6 R 1.5 (4DRs/4x2Gb/z990)TS7720 VEA/CS7/XS7 R 1.5 (7DRs/4x4Gb/z10)TS7740 V06/CC7/CX7 R 1.6 (4DRs/4x4Gb/z10)TS7720 VEA/CS7/XS7 R 1.6 (7DRs/4x4Gb/z10)TS7740 V06/CC8/CX7 R 1.7 (4DRs/4x4Gb/z10)TS7720 VEA/CS8/XS7 R 1.7 (7DRs/4x4Gb/z10)

TS7720 VEA/CS8/XS7 R 1.7 (19DRs/4x4Gb/z10)TS7740 V07/CC8/CX7 R 2.0 (4DRs/4x4Gb/z10)TS7720 VEB/CS8/XS7 R 2.0 (7DRs/4x4Gb/z10)

TS7720 VEB/CS8/XS7 R 2.0 (19DRs/4x4Gb/z10)TS7740 V07/CC8/CX7 R 2.1 (4DRs/4x4Gb/z10)TS7720 VEB/CS8/XS7 R 2.1 (7DRs/4x4Gb/z10)

TS7720 VEB/CS9/XS9 R 3.0 (10-DRs/4x4Gb/z10)TS7740 V07/CC9/CX9 R 3.0 (3DRs/4x4Gb/z10)

TS7740 V07/CC9/CX9 R 3.1 (3DRs/4x1x8Gb/z10)TS7720 VEB/CS9/XS9 R 3.1 (10-DRs/8x8Gb/z10)

TS7720 VEB/CS9*/XS9 R 3.1 (10-DRs/8x8Gb/z10)TS7720 VEB/CS9*/XS9 R 3.1 (26-DRs/8x8Gb/z10)TS7720 VEB/CS9*/XS9 R 3.1 (42-DRs/8x8Gb/z10)


Standalone VTS/TS7700 Read Hit Performance History

Read Hit



TS7700 Copy Performance Evolution

Figures 3 through 4 display deferred copy rates. Data rate over the grid links are of compressed data. In each of the following runs, a deferred copy mode run was ended following several terabyte (TB) of data being written to the active cluster(s). In the subsequent hours, copies took place from the source cluster to the target cluster. There was no other TS7700 activity during the deferred copy except for premigration if the source or target cluster was a TS7740.

The 8Gb FICON requires an additional 16GB of memory (total 32GB), which accounts for the copy performance improvement in R 3.1 with 8Gb FICON.

Figure 3. Two-way TS7700 Single-direction Copy Bandwidth.

0 200 400 600 800 1000 1200

Single-direction Copy MB/s (compressed)

TS7740 R 1.6 (V06/CC7/4 DRs/8GB RAM /7 premig drives/2x1Gb links)

TS7720 R 1.6 (VEA/CS7/7 DRs/8GB RAM /2x1Gb links)








TS7720 R 2.0 (VEB/CS8/7 DRs/16GB RAM /2x1Gb links)



















Two-Way TS7700 Single-direction Copy Performance History

Sustained Copy Peak Copy



Figure 4. Two-way TS7700 Dual-direction Copy Bandwidth

0 200 400 600 800 1000 1200 1400

Dual-direction Copy MB/s (compressed)

TS7740 R 1.6 (V06/CC7/4 DRs/8GB RAM/7 premig drives/2x1Gb links)TS7720 R 1.6 (VEA/CS7/7 DRs/8GB RAM/2x1Gb links)

TS7740 R 1.7 (V06/CC8/4 DRs/8GB RAM/7 premig drives/2x1Gb links)TS7720 R 1.7 (VEA/CS8/7 DRs/8GB RAM/2x1Gb links)

TS7720 R 1.7 (VEA/CS8/19 DRs/8GB RAM/2x1Gb links)TS7740 R 2.0 (V07/CC8/4 DRs/16GB RAM/6 premig drives/2x1Gb links)TS7740 R 2.0 (V07/CC8/4 DRs/16GB RAM/6 premig drives/4x1Gb links)

TS7740 R 2.0 (V07/CC8/4 DRs/16GB RAM/11 premig drives/4x1Gb links)TS7740 R 2.0 (V07/CC8/4 DRs/16GB RAM/6 premig drives/2x10Gb links)

TS7720 R 2.0 (VEB/CS8/7 DRs/16GB RAM/2x1Gb links)TS7720 R 2.0 (VEB/CS8/19 DRs/16GB RAM/2x1Gb links)TS7720 R 2.0 (VEB/CS8/7 DRs/16GB RAM/4x1Gb links)

TS7720 R 2.0 (VEB/CS8/19 DRs/16GB RAM/4x1Gb links)TS7720 R 2.0 (VEB/CS8/7 DRs/16GB RAM/2x10Gb links)

TS7720 R 2.0 (VEB/CS8/19 DRs/16GB RAM/2x10Gb links)TS7740 R 2.1 (V07/CC8/4 DRs/16GB RAM/10 premig drives/4x1Gb links)

TS7740 R 2.1 (V07/CC8/4 DRs/16GB RAM/10 premig drives/2x10Gb links)TS7720 R 2.1 (VEB/CS8/7 DRs/16GB RAM/4x1Gb links)

TS7740 R 3.0 (V07/CC9/3 DRs/16GB RAM/10 premig drives/4x1Gb links)TS7740 R 3.0 (V07/CC9/3 DRs/16GB RAM/10 premig drives/2x10Gb links)


TS7720 R 3.0 (VEB/CS9/10 DRs/16GB RAM/2x10Gb links)TS7740 R 3.1 (V07/CC9/3 DRs/32GB RAM/10 premig drives/4x1Gb links)

TS7740 R 3.1 (V07/CC9/3 DRs/32GB RAM/10 premig drives/2x10Gb links)TS7720 R 3.1 (VEB/CS9/10 DRs/32GB RAM/4x1Gb links)


Two-Way TS7700 Dual-direction Copy Performance History

Sustained Copy Peak Copy



Hardware Configuration

The following hardware was used in performance measurements. Performance workloads are driven from IBM System z10 host with four or eight 8Gb FICON channels, except for the 4Gb channel versus 8Gb channel tests which used 4Gb z10 host channels.

Standalone Hardware Setup

Grid Hardware Setup

Notes: The following conventions are used in this paper:

TS7700 Model Drawer count

Cache size Tape Drives

TS7720 VEB 3956 CS9 1 23.86 TB N/A

TS7720 VEB 3956 CS9/XS9 2 47.85 TB N/A

TS7720 VEB 3956 CS9/XS9 3 71.84 TB N/A

TS7720 VEB 3956 CS9/XS9 6 143.82 TB N/A

TS7720 VEB 3956 CS9/XS9 10 239.78 TB N/A

TS7720 VEB 3956 CS9/XS9 26 623.78 TB N/A

TS7720 VEB 3956 CS9/XS9 42 1007.78 TB N/A

TS7740 V07 3956 CC9/CX9 1 9.45 TB 12 TS1140

TS7740 V07 3956 CC9/CX9 2 19.03 TB 12 TS1140

TS7740 V07 3956 CC9/CX9 3 28.60 TB 12 TS1140

TS7700 Model Drawer count

Cache size Tape Drives Grid links (Gb)

TS7720 VEB 3956 CS9/XS9 10 239.78 TB N/A 4x1 or 2x10

TS7720 VEB 3956 CS9/XS9 26 623.78 TB N/A 4x1 or 2x10

TS7740 V07 3956 CC9/CX9 3 28.60 TB 12 TS1140 4x1 or 2x10

Binary Decimal

Name Symbol Values in Bytes Name Symbol Values in Bytes

kibibyte KiB 210

kilobyte KB 103

mebibyte MiB 220

megabyte MB 106

gibibyte GiB 230

gigabyte GB 109

tebibyte TiB 240

terabyte TB 1012

pebibyte PiB 250

petabyte PB 1015



TS7700 Performance Overview

Performance Workloads and Metrics

Performance shown in this paper has been derived from measurements that generally attempt to simulate common user environments, namely a large number of jobs writing and/or reading multiple tape volumes simultaneously. Unless otherwise noted, all of the measurements were made with 128 simultaneously active virtual tape jobs per active cluster. Each tape job was writing or reading 800 MiB of uncompressed data using 32 KiB blocks and QSAM BUFNO=20, using data that compresses within the TS7700 at 2.66:1. Measurements were made with eight 8-gigabit (Gb) FICON channels on a z10 host. All runs begin with the virtual tape subsystem inactive.

Unless otherwise stated, all runs were made with default tuning values (DCOPYT=125, DCTAVGTD=100, PMPIOR=1600, PMTHLVL=2000, ICOPYT=ENABLED, Reclaim disabled, Number of premigration drives per pool=10). Refer to the IBM® Virtualization Engine TS7700 Series Best Practices - Understanding, Monitoring and Tuning the TS7700 Performance white paper for detailed description of different tuning settings.

Types of Throughput

Because the TS7720 is a disk-cache only cluster. The read and write data rates have been found to be fairly consistent throughout a given workload. Because the TS7740 contains physical tapes to which the cache data will be periodically written and read, the TS7740 has been found to exhibit four basic throughput rates: peak write, sustained write, read-hit, and recall. These four rates are described below.

Peak and Sustained Write Throughput.

For all TS7740 measurements, any previous workloads have been allowed to quiesce with respect to pre-migration to back tape and replication to other clusters in the grid. In other words, the test is started with the grid in an idle state. Starting with this initial idle state, data from the host is first written into the TS7740 disk cache with little if any premigration activity taking place. This allows for a higher initial data rate, and is termed the “peak” data rate. Once a pre-established threshold is reached of non-premigrated compressed data, the amount of premigration is increased, which can reduce the host write data rate. This threshold is called the premigration priority threshold, and has default value of 1600 gigabytes (GB). When a further threshold of non-premigrated compressed data is reached, the incoming host activity is actively throttled to allow for increased premigration activity. This throttling mechanism operates to achieve a balance between the amount of data coming in from the host and the amount of data being copied to physical tape. The resulting data rate for this mode of behavior is called the “sustained” data rate, and could theoretically continue on forever, given a constant supply of logical and physical scratch tapes. This

Metrics and Workloads



second threshold is called the premigration throttling threshold, and has a default value of 2000 gigabytes (GB). These two thresholds can be used in conjunction with the peak data rate to project the duration of the peak period. Note that both the priority and throttling thresholds can be increased or decreased via a host command line request.

Read-hit and Recall Throughput

Similar to write activity, there are two types of TS7740 read performance: “read-hit” (also referred to as “peak”) and “recall” (also referred to as “read-miss”). A read hit occurs when the data requested by the host is currently in the local disk cache. A recall occurs when the data requested is no longer in the disk cache and must be first read in from physical tape. Read-hit data rates are typically higher than recall data rates.

These two read performance metrics, along with peak and sustained write performance are sometimes referred to as the “four corners” of virtual tape performance. The following charts in the paper show three of these corners:

1. peak write 2. sustained write 3. read hit Recall performance is dependent on several factors that can vary greatly from installation to installation, such as number of physical tape drives, spread of requested logical volumes over physical volumes, location of the logical volumes on the physical volumes, length of the physical media, and the logical volume size. Because these factors are hard to control in the laboratory environment, recall is not part of lab measurement.



Grid Considerations

Up to four TS7700 clusters can be linked together to form a Grid configuration. Five- and six-way grid configurations are available via iRPQ. The connection between these clusters is provided by two 1-Gb TCP/IP links (default). Four 1-Gb links or two 10-Gb links options are also available. Data written to one TS7700 cluster can be optionally copied to the one or more other clusters in the grid.

Data can be copied between the clusters in either deferred, RUN (also known as “Immediate”), or sync mode copy. When using the RUN copy mode the rewind-unload response at job end is held up until the received data is copied to all peer clusters with a RUN copy consistency point. In deferred copy mode data is queued for copying, but the copy does not have to occur prior to job end. Deferred copy mode allows for a temporarily higher host data rate than RUN copy mode because copies to the peer cluster(s) can be delayed, which can be useful for meeting peak workload demands. Care must be taken, however, to be certain that there is sufficient recovery time for deferred copy mode so that the deferred copies can be completed prior to the next peak demand. Whether delay occurs and by how much is configurable through the Library Request command. In sync mode copy, data synchronization is up to implicit or explicit sync point granularity across two clusters within a grid configuration. In order to provide a redundant copy of these items with a zero recovery point objectives (RPO), the sync mode copy function will duplex the host record writes to two clusters simultaneously.

Grid Considerations



TS7700 Basic Performance

The following sets of graphs show basic TS7700 bandwidths. The graphs in Figures 5 and 6 show single cluster, standalone configurations. Unless otherwise stated, the performance metric shown in these and all other data rate charts in this paper is host-view (uncompressed) MB/sec.

TS7720 Standalone Performance

Standalone TS7720 R 3.1 Performance

0

500

1000

1500

2000

2500

Write 1:1 Read 1:1 Mixed 1:1 Write 2.66:1 Read 2.66:1 Mixed 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

VEB/CS9*/1 DRs/8x8Gb VEB/CS9*/2 DRs/8x8Gb VEB/CS9*/3 DRs/8x8GbVEB/CS9*/6 DRs/8x8Gb VEB/CS9/10 DRs/8x8Gb VEB/CS9*/10 DRs/8x8GbVEB/CS9*/26 DRs/8x8Gb VEB/CS9*/42 DRs/8x8Gb

Figure 5. TS7720 Standalone Maximum Host Throughput. All runs were made with 128 concurrent jobs, each job writing 800 MiB (uncompressed) using 32KiB blocks, QSAM BUFNO = 20, using eight 8Gb (8x8Gb) FICON channels from a z10 LPAR.

Notes:

• CS9* -- the new higher-performance 3TB drive (yellow bar as compared to

pink bar).

• Mixed workload refers to 50% write plus 50% read hit workload driven from the host. The resulting data rate seen from TS7720 is not 50-50 read/write mixed; normally it is more read than write.

TS7720

Standalone Maximum Host Throughput



TS7740 Standalone Performance

Standalone TS7740 R 3.1 Performance

0

500

1000

1500

2000

2500

Pk Wr 1:1 St. Wr 1:1 Read 1:1 Pk Mx 1:1 St. Mx 1:1 Pk Wr

2.66:1

St. Wr

2.66:1

Read

2.66:1

Pk Mx

2.66:1

St. Mx

2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

V07/CC9/1 DRs/12 E07s/8x8Gb V07/CC9/2 DRs/12 E07s/8x8GbV07/CC9/3 DRs/12 E07s/8x8Gb

Figure 6. TS7740 Standalone Maximum Host Throughput. All runs were made with 128 concurrent jobs, each job writing 800 MiB (uncompressed) using 32KiB blocks, QSAM BUFNO = 20, using eight 8Gb (8x8Gb) FICON channels from a z10 LPAR.

Notes: Mixed workload refers to 50% write plus 50% read hit workload driven from the host. The resulting data rate seen from TS7740 is not 50-50 read/write mixed; normally it is more read than write.

TS7740 Standalone Maximum Host Throughput



TS7700 Grid Performance

Figures 7, 8, 10 through 12, 14, 16, 18, 20, 22, and 24 display the performance for TS7700 grid configurations.

For these charts “D” stands for deferred copy mode, “S” stands for sync mode copy and “R” stands for RUN (immediate) copy mode. For example, in Figure 5, RR represents RUN for cluster 0, and RUN for cluster 1. SS refers to synchronous copies for both clusters.

All measurements for these graphs were made at zero or near-zero distance between clusters.

Two-way TS7700 Grid with Single Active Cluster Performance

Two-way TS7720 R 3.1 Performance

(Single Active Cluster)

0

500

1000

1500

2000

2500

3000

3500

DD Write 2.66:1 SS Write 2.66:1 RR Write 2.66:1 Read Hit 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

VEB/CS9/10 Dr/2x2x8Gb/4x1Gb VEB/CS9*/10 Dr/4x1x8Gb/4x1Gb

VEB/CS9*/10 Dr/8x8Gb/4x1Gb VEB/CS9/10 Dr/8x8Gb/4x1Gb

VEB/CS9/10 Dr/8x8Gb/2x10Gb VEB/CS9/26 Dr/8x8Gb/2x10Gb

Figure 7. Two-way TS7720 Single Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 128 concurrent jobs. Each job writing or reading 800 MiB (300MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight or four 8Gb FICON channels from a z10 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Notes:

• 8x8Gb – eight 8Gb FICON channels (four cards, dual ports per card)

• 4x1x8Gb – four 8Gb FICON channels (four cards, single port per card)


Two-way TS7700 Grid Single Active Maximum Host Throughput

TS7700 Grid Maximum Host Throughput





0

500

1000

1500

2000

2500

DD Peak

Write

2.66:1

DD Sust.

Write

2.66:1

SS Peak

Write

2.66:1

SS Sust.

Write

2.66:1

RR Peak

Write

2.66:1

RR Sust.

Write

2.66:1

Read Hit -

2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

V07/CC9/3 DRs/4x1x8Gb/2x10G V07/CC9/3 DRs/8x8Gb/4x1Gb

V07/CC9/3 DRs/8x8Gb/2x10Gb

Figure 8. Two-way TS7740 Single Active Maximum Host Throughput. All runs were made with 128 concurrent jobs. Each job writing or reading 800 MiB (300MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight or four 8Gb FICON channels from a z10 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Notes:





Figure 9. Two-way TS7700 Hybrid Grid H1

Two-way TS7700 R 3.1 Hybrid H1 Performance


0

500

1000

1500

2000

2500

DD Write

2.66:1

SS Peak Write

2.66:1

SS Sust. Write

2.66:1

RR Peak Write

2.66:1

RR Sust. Write

2.66:1

Read Hit

2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

VEB/CS9/10 DRs - V07/CC9/3 DRs - 8x8Gb/4x1Gb



Figure 10. Two-way TS7700 Hybrid H1 Single Active Maximum Host Throughput. All runs were made with 128 concurrent jobs. Each job writing or reading 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.



Two-way TS7700 Grid with Dual Active Cluster Performance


(Dual Active Clusters)

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

DD Write 2.66:1 SS Write 2.66:1 RR Write 2.66:1 Read Hit 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

VEB/CS9/10 Dr/2x2x8Gb/4x1Gb VEB/CS9*/10 Dr/4x1x8Gb/4x1Gb

VEB/CS9*/10 Dr/8x8Gb/4x1Gb VEB/CS9/10 Dr/8x8Gb/4x1Gb

VEB/CS9/10 Dr/8x8Gb/2x10Gb VEB/CS9/26 Dr/8x8Gb/2x10Gb

Figure 11. Two-way TS7720 Dual Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing 800 MiB (300MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight or four 8Gb FICON channels from a z10 LPAR. Read tests are driven from two z10 LPARs. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125. Notes:



• CS9* -- the new high-performance 3TB drive.

Two-way TS7700 Grid Dual Active Maximum Host Throughput





0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

DD Peak

Write

2.66:1

DD Sust.

Write

2.66:1

SS Peak

Write

2.66:1

SS Sust.

Write

2.66:1

RR Peak

Write

2.66:1

RR Sust.

Write

2.66:1

Read Hit -

2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

V07/CC9/3 DRs/4x1x8Gb/2x10G V07/CC9/3 DRs/8x8Gb/4x1Gb

V07/CC9/3 DRs/8x8Gb/2x10Gb

Figure 12. Two-way TS7740 Dual Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Read tests are driven from two z10 LPARs. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.



Figure 13. Two-way TS7700 Hybrid Grid H2

Two-way TS7700 Hybrid H2 R 3.1 Performance


0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

DD Peak Write

2.66:1

DD Sust. Write

2.66:1

SS Peak Write

2.66:1

SS Sust. Write

2.66:1

RR Peak Write

2.66:1

RR Sust. Write

2.66:1

Read Hit 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)




Figure 14. Two-way TS7700 Hybrid H2 Dual Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Read tests are driven from two z10 LPARs. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.



Three-way TS7700 Grid with Dual Active Cluster Performance

Figure 15. Three-way TS7700 Hybrid Grid H3

Three-way TS7700 Hybrid H3 R 3.1 Performance


0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

RND/NRD Write 2.66:1 SSD Write 2.66:1 RRD Write 2.66:1 Read Hit 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)




Figure 16. Three-way TS7700 Hybrid H3 Dual Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Read tests are driven from two z10 LPARs. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.

Three-way TS7700 Grid Dual Active Maximum Host Throughput



.

Figure 17. Three-way TS7700 Hybrid Grid H4

Three-way TS7700 Hybrid H4 R 3.1 Performance


0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

DDD Peak

Write 2.66:1

DDD Sust. Write

2.66:1

SSD Peak

Write 2.66:1

SSD Sust. Write

2.66:1

RRD Peak

Write 2.66:1

RRD Sust. Write

2.66:1

Read Hit 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)




Figure 18. Three-way TS7700 Hybrid H4 Dual Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Read tests are driven from two z10 LPARs. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.



Four-way TS7700 Grid with Dual Active Cluster Performance

Figure 19. Four-way TS7700 Hybrid Grid H5

Four-way TS7700 Hybrid H5 R 3.1 Performance


0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

RNDD/NRDD Peak Write 2.66:1 Read Hit 2.66:1

Ho

st M

B/s

(U

nco

mp

ressed

)




Figure 20. Four-way TS7700 Hybrid H5 Dual Active Maximum Host Throughput. All runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Read tests are driven from two z10 LPARs. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125..

Four-way TS7700 Hybrid Grid Dual Active Maximum Host Throughput






0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

DDDD Peak

Write 2.66:1

DDDD Sust.

Write 2.66:1

SSDD Peak

Write 2.66:1

SSDD Sust.

Write 2.66:1

RRDD Peak

Write 2.66:1

RRDD Sust.

Write 2.66:1

Read Hit 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)




Figure 22. Four-way TS7700 Hybrid H7 Dual Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 256 concurrent jobs (128 jobs per active cluster). Each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Read tests are driven from two z10 LPARs. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.





(Quadruple Active Clusters)

0

500

1000

1500

2000

2500

3000

3500

4000

4500

SSDD-DDSS Peak Write

2.66:1

SSDD-DDSS Sust. Write

2.66:1

RRDD-DDRR Peak Write

2.66:1

RRDD-DDRR Sust. Write

2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)




Figure 24. Four-way Hybrid H7 Quadruple Active Maximum Host Throughput. Unless otherwise stated, all runs were made with 256 concurrent jobs per z10 LPAR (each LPAR drives two clusters). Each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32 KiB block size, QSAM BUFFNO = 20, using eight 8Gb FICON channels from a z10 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup. DCT=125.

Four-way TS7700 Hybrid H7 Quadruple Active Maximum Host Throughput



Additional Performance Metrics

Performance vs. Volume size

Figure 25 shows data rates on 2-way TS7700 (single active cluster) using RR or SS copy mode with different volume size. For volume sizes that are 4 GiB or larger, the TS7720 internal immediate mode copy throttle mechanism can slightly impact the overall performance of RR runs. Smaller volumes may be more ideal, or the throttling can be disabled.

Two-way TS7720 R 3.1 Performance vs. Volume Size


0

500

1000

1500

SS Write 2.66:1 RR Write 2.66:1

Ho

st

MB

/s (

Un

co

mp

res

se

d)

300 MiB compressed 1000 MiB compressed

2000 MiB compressed 4000 MiB compressed

Figure 25. TS7720 VEB 3956 CS9/XS9/10 drawers/4x1Gb links (single active cluster) RR and SS performance vs. volume size. All runs were made with 128 concurrent jobs, using 32KB blocks, QSAM BUFNO = 20, and eight 8Gb FICON channels from a z10 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup.

Two-way TS7720 Single-active RR/SS Data Rates vs. Volume Size



Figure 26 shows data rates on 2-way TS7700 (dual active cluster) using RR or SS copy mode with different volume sizes. For volume sizes that are 4 GiB or larger, the TS7700’s internal immediate mode copy throttle may reduce the overall performance of the RR runs, which can result in irregular host throughput. In the figure 26, the RR run with 4 GiB volume size was completed with the immediate throttle turned off given the inter-cluster latency in the laboratory setup was low enough in which immediate throttle was not required. If latency and/or performance is non-ideal, it may enter the immediate-deferred state when the immediate throttling is disabled. Please work with your specialist in deciding whether immediate mode throttling can be disabled within your configuration.

Two-way TS7720 R 3.1 Performance vs. Volume Size


0

500

1000

1500

2000

2500

SS Write 2.66:1 RR Write 2.66:1Ho

st

MB

/s (

Un

co

mp

res

se

d)

300 MiB compressed1000 MiB compressed2000 MiB compressed4000 MiB compressed (turned off immed throttle)

Figure 26. TS7720 VEB 3956 CS9/XS9/10 drawers/4x1Gb links (dual active clusters) RR and SS performance vs. volume size. All runs were made with 128 concurrent jobs, using 32KB blocks, QSAM BUFNO = 20, and eight 8Gb FICON channels from a z10 LPAR. Clusters are located at zero or near zero distance to each other in laboratory setup.

Two-way TS7720 Dual-active RR/SS Data Rates vs. Volume Size



Performance vs. Host Channels

Figure 27 shows data rates on a standalone TS7700 VEB/CS9/10 drawers/8Gb FICON with workload driven from a z10 host supporting 8Gb FICON vs. workload driven from a z10 host only supporting 4Gb FICON. There is no noticeable performance difference seen with dual port per TS7700 8Gb FICON card when workloads are driven using 4Gb host channels or 8Gb host channels. This reveals how the 8Gb FICON adapter within the TS7700 is the limiting component and not the channel speed when both ports are used per 8Gb FICON card on the TS7700.

Standalone TS7720 R 3.1 Performance vs. Host Channel

(Dual ports per card)

0

500

1000

1500

2000

2500


Ho

st

MB

/s (

Un

co

mp

res

se

d)

VEB/CS9/10 DRs/8x8Gb<->8x8Gb host channels

VEB/CS9/10 DRs/8x8Gb<->8x4Gb host channels

Figure 27. TS7720 Standalone Maximum Host Throughput. All runs were made with 128 concurrent jobs, each job writing 800 MiB (uncompressed) using 32KiB blocks, QSAM BUFNO = 20.



Figure 28 shows the performance improvement when upgrading the TS7700 from four 4Gb FICON cards to 8Gb FICON (single port used per card) adapters. The comparison is against a host infrastructure that remains 4Gb and one which also upgrades to 8Gb FICON on the z10 host. Even without moving to a complete 8Gb FICON infrastructure on the z10 host, the new 8Gb FICON adapters out-perform the previous generation 4Gb adapter cards.

Standalone TS7720 Performance

(Single port per card)

0

500

1000

1500

2000

2500


Ho

st

MB

/s (

Un

co

mp

res

se

d)

R 3.0 VEB/CS9/10 DRs/4x4Gb<->4x4Gb host channels *

R 3.1 VEB/CS9/10 DRs/4x1x8Gb<->4x4Gb host channels

R 3.1 VEB/CS9/10 DRs/4x1x8Gb<->4x8Gb host channels **

Figure 28. TS7720 Standalone Maximum Host Throughput. All runs were made with 128 concurrent jobs, each job writing 800 MiB (uncompressed) using 32KiB blocks, QSAM BUFNO = 20.

* the 4Gb FICON data are taken from the R 3.0 Performance White Paper.

** the four 8Gb host channels should be located on different cards on the z10 host. If two 8Gb host channels are located on the same card on the z10 host, the write rate will be about 13% lower, and the read rate will be about 6% lower.



Performance at Distance

Grid performance has been seen to vary depending on:

1. distance between clusters, 2. quality of links, 3. speed of links (1Gb or 10 Gb), 4. number of utilized TCP/IP links (up to 4 for 1Gb link and up to 2 for 10 Gb

link), 5. percentage of link bandwidth available, 6. Type of I/O activity (write, read, or copy). The following setup was used to assess the deferred copy and sync mode copy performance vs. different inter-cluster distances and packet loss percentages in the laboratory:

• Two 10Gb setup -- Two TS7720 VEB/CS9/26 drawer/8x8Gb FICON were connected using two 10Gb TCP/IP links

• Four 1Gb setup -- Two TS7720 VEB/CS9/10 drawer/8x8Gb FICON were connected using two 1Gb TCP/IP links

• Two Ethernet Network Emulators were placed between the two clusters

• Time delay and/or packet loss were applied to both directions to simulate different latencies associated with distance as well as packet loss scenarios.



Sync Mode Copy over Distance and Packet Losses

Figure 29 shows the host rate when sync mode copy is configured between two TS7720 3956 VEB/CS9/26 drawers/8x8Gb FICON channels/2x10Gb links (single active) with different distances/packet losses. Performance degrades as distance increases (1ms time delay is estimated to be roughly 100 miles in distance).

Two-way TS7720 (VEB/CS9/10-DR/2x10G) 1-way Sync Mode

Copy Performance vs. 1-way Latency and Packet Loss

0

500

1000

1500

2000

2500

0 20 40 60 80

One-way Latency (ms)

Ho

st

MB

/s (

Un

co

mp

res

se

d)

0.000% Packet Loss

0.003% Packet Loss

0.030% Packet Loss

Figure 29. Two-wayTS7720 VEB 3956 CS9/XS9/26 drawers/8x8Gb FICON/2x10Gb link (single active cluster) sync mode copy performance vs latency and/or packet loss. All runs were made with 128 concurrent jobs, each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32KB blocks, QSAM BUFNO = 20, and eight 8Gb FICON channels from a z10 LPAR.

Notes: Packet loss refers to packet drop on the network (eg. 0.03% packet loss implies one packet drop per 3333 packets). Packet loss is not the same as the re-transmit rate.

TS7720 VEB/CS9/XS9 (26-Drawers/ 8x8Gb FICON channels/2x10Gb links) Single-direction Sync Mode Copy



Figure 30 shows the host rate when sync mode copy is configured between two TS7720 3956 VEB/CS9/10 drawers/8x8Gb FICON channels/4x1Gb links (single active) with different distances/packet losses. Performance degrades as distance increases (1ms time delay is estimated to be roughly 100 miles in distance).

Two-way TS7720 (VEB/CS9/10-DR/4x1Gb) 1-way Sync Mode Copy

Performance vs. 1-way Latency and Packet Loss

0

200

400

600

800

1000

1200

1400

0 10 20 30 40 50 60 70 80


Ho

st

MB

/s (

Un

co

mp

res

se

d)

0.000% Packet Loss

0.003% Packet Loss

0.030% Packet Loss

Figure 30. Two-wayTS7720 VEB 3956 CS9/XS9/10 drawers/8x8Gb FICON/4x1Gb link (single active cluster) sync mode copy performance vs latency and/or packet loss. All runs were made with 128 concurrent jobs, each job writing 10.64 GiB (4 GiB volumes @ 2.66:1 compression) using 32KB blocks, QSAM BUFNO = 20, and eight 8Gb FICON channels from a z10 LPAR.

Notes: Packet loss refers to packet drop on the network (eg. 0.03% packet loss implies one packet drop per 3333 packets). Packet loss is not the same as the re-transmit rate.

TS7720 VEB/CS9/XS9 (10-Drawers/ 8x8Gb FICON channels/4x1Gb links) Single-direction Sync Mode Copy



Deferred Copy Performance over Distance and/or Packet Losses

Figure 31 shows the single-direction deferred copy rate between two TS7720 3956 VEB CS9 and twenty-five XS9 drawer configurations with eight 8Gb FICON channels. The two TS7720 are connected via 2x10Gb links. This provides an example of how latency and packet loss can affect replication rates.

In the run, host I/O was driven to one TS7720 (active cluster) in deferred copy mode. After tens of terabytes (TB) of uncopied data were queued at the active cluster, the workload was stopped and the only activity was the replication from the one TS7720 to the second TS7720. Different one-way time delay values and packet loss percentages were applied to both directions.

Two-way TS7720 (VEB/CS9/26-DR/2x10G) 1-way Deferred Copy


0

200

400

600

800

1000

1200

0 20 40 60 80


Co

py

MB

/s (

Co

mp

res

se

d)

0.000% Packet Loss

0.003% Packet Loss

0.030% Packet Loss

Figure 31. Two-wayTS7720 VEB 3956 CS9/XS9/26 drawers/2x10Gb link (single active cluster) deferred copy performance vs latency and/or packet loss. All runs were made with 128 concurrent jobs, each job writing 800 MiB (300 MiB volumes @ 2.66:1 compression) using 32KB blocks, QSAM BUFNO = 20, and eight 8Gb FICON channels from a z10 LPAR.

Notes: Packet loss refers to packet drop on the network (eg. 0.03% packet loss implies one packet drop per 3333 packets). The packet loss is not the same as the re-transmit rate.

TS7720 VEB/CS9/XS9 (26-Drawers/ 8x8Gb FICON channels/2x10Gb links) Single-direction Deferred Copy over Distance and/or Packet Loss



Figure 32 shows the single-direction deferred copy rate between two TS7720 3956 VEB CS9 and nine XS9 drawer configurations with eight 8Gb FICON channels. The two TS7720 are connected via 4x1Gb links. This provides an example of how latency and packet loss can affect replication rates.

In the run, host I/O was driven to one TS7720 (active cluster) in deferred copy mode. After tens of terabytes (TB) of uncopied data were queued at the active cluster, the workload was stopped and the only activity was the replication from the one TS7720 to the second TS7720. Different one-way time delay values and packet loss percentages were applied to both directions.

Two-way TS7720 (VEB/CS9/10-DR/4x1Gb) 1-way Deferred Copy


0

50

100

150

200

250

300

350

400

450

500

0 10 20 30 40 50 60 70 80


Co

py M

B/s

(C

om

pre

ssed

)

0.000% Packet Loss

0.003% Packet Loss

0.030% Packet Loss

Figure 32. Two-wayTS7720 VEB 3956 CS9/XS9/26 drawers/2x10Gb link (single active cluster) deferred copy performance vs latency and/or packet loss. All runs were made with 128 concurrent jobs, each job writing 10.64 GiB ( 4 GiB volumes @ 2.66:1 compression) using 32KB blocks, QSAM BUFNO = 20, and eight 8Gb FICON channels from a z10 LPAR.

Notes: Packet loss refers to packet drop on the network (eg. 0.03% packet loss implies one packet drop per 3333 packets). The packet loss is not the same as the re-transmit rate.

TS7720 VEB/CS9/XS9 (10-Drawers/ 8x8Gb FICON channels/4x1Gb links) Single-direction Deferred Copy over Distance and/or Packet Loss



Performance Tools

Batch Magic

This tool is available to IBM representatives and Business Partners to analyze SMF data for an existing configuration and workload, and project a suitable TS7700 configuration.

BVIRHIST plus VEHSTATS

BVIRHIST requests historical statistics from a TS7700, and VEHSTATS produces the reports. The TS7700 keeps the last 90 days of statistics. BVIRHIST provides a means for users to save statistics for periods longer than 90 days.

Performance Analysis Tools A set of performance analysis tools is available on Techdocs that utilizes the data generated by VEHSTAT. Provided are spreadsheets, data collection requirements, and a 90 day trending evaluation guide to assist in the evaluation of the TS7700 performance. Spreadsheets for a 90 day, one week, and a 24 hour evaluation are provided. http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4717 Also, on the Techdocs site is a webinar replay that teaches you how to use the performance analysis tools. http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4872

BVIRPIT plus VEPSTATS

BVIRPIT requests point-in-time statistics from a TS7700, and VEPSTATS produces the reports. Point-in-time statistics cover the last 15 seconds of activity and give a snapshot of the current status of drives and volumes.

The above tools are available at one of the following web sites:

External users: http://public.dhe.ibm.com/storage/tapetool/ Internal users: ftp://submit.boulder.ibm.com/download/tapetool/ Business Partners tape tools: http://testcase-yellow.boulder.ibm.com

Performance Aids



Conclusions

The TS7700 Virtualization Engine provides significant performance and increased capacity. Release 3.1 introduces 8Gb FICON and doubles the supported FICON connections from four to eight. The 8Gb FICON feature comes with an additional 16GB memory. The 8Gb FICON combined 32GB memory dramatically improves the TS7700 performance, up to 120% in some cases. The TS7700 architecture will continue to provide a base for product growth in both performance and functionality.

Conclusions



Acknowledgements

The authors would like to thank Joseph Swingler, Jim Fisher, Donald Denning, Lawrence Fuss, Erika Dawson, and Anthony Lambert for their review comments and insight, and also to Eileen Maroney and Jim Fisher for distributing the paper.

Finally, the authors would like to thank Dennis Martinez, James F. Tucker, Harold Koeppel, Douglas Clem, Michael Frick, William Hengen, and Scott Ratzloff for hardware/zOS/network support.

Acknowledgements



© International Business Machines Corporation 2013

IBM Systems Group

5600 Cottle Road,

San Jose, CA 95193

Printed in the United States of America

3-14

All Rights Reserved

IBM, the IBM logo, System Storage, System z, zOS, TotalStorage, Virtualization Engine, DFSMSdss, DFSMShsm, ESCON and FICON are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

Other company, product and service names may be trademarks or service marks of others.

Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice.

This information could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or programs(s) at any time without notice.

Performance data for IBM and non-IBM products and services contained in this document were derived under specific operating and environmental conditions. The actual results obtained by any party implementing such products or services will depend on a large number of factors specific to such party’s operating environment and may vary significantly. IBM makes no representation that these results can be expected or obtained in any implementation of any such products or services.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.



THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED “AS IS” WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR INFRINGEMENT.

IBM shall have no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided.

ibm virtualization engine ts7720 and ts7740 … virtualization engine ts7700 release 3.1 performance...

Documents