introduction · web view2015/06/26 · ibm z13 and ficon express16s channel performance june 26,...

IBM z13 and FICON Express16S Channel Performance

June 26, 2015

Brian MurphyDinesh Kumar

zEnterprise I/O Performance

IBM System z13 I/O and High Performance FICON for System z Channel Performance

Table of ContentsTable of Contents......................................................................................................................................2Introduction.................................................................................................................................................3The High Performance FICON for System z journey:................................................................3

Figure 1.....................................................................................................................................................4High Performance FICON for System z (zHPF) architecture performance features....6

Figure 2.....................................................................................................................................................7zHPF Exploitation.....................................................................................................................................8FICON Express16S channel performance using zHPF and FICON protocols.................9

Figure 3.....................................................................................................................................................9Figure 4...................................................................................................................................................10Figure 5...................................................................................................................................................11Figure 6...................................................................................................................................................12Figure 7...................................................................................................................................................13Figure 8...................................................................................................................................................14Figure 9...................................................................................................................................................15

OLTP and Commercial Batch Workload Performance.............................................................15Figure 10................................................................................................................................................16

FEx16S Channel Performance using zHPF and FICON Protocols at up to 100km of Distance......................................................................................................................................................17

Figure 11................................................................................................................................................18Figure 12................................................................................................................................................19Table 1.....................................................................................................................................................20Figure 13................................................................................................................................................20Figure 14................................................................................................................................................21

zHPF Extended Distance II.................................................................................................................21Figure 15................................................................................................................................................22Figure 16................................................................................................................................................22

zHPF fields on the RMF Channel Activity report.......................................................................22Figure 17................................................................................................................................................23

FICON Express16S card and I/O Subsystem (IOSS) performance with zHPF..............24Figure 18................................................................................................................................................24Figure 19................................................................................................................................................25Figure 20................................................................................................................................................26

z13 SAP capacity.....................................................................................................................................27Figure 21................................................................................................................................................28

Summary.....................................................................................................................................................29Acknowledgments...............................................................................................................................30

2


IntroductionThis technical paper was developed to help IBM field sales specialists and technical representatives understand the I/O performance characteristics of the FICON Express16S channels available on the IBM zEnterprise 13 (z13) servers which support both High Performance FICON for System z (zHPF) and FICON protocols simultaneously using the channel path identifier (CHPID) type FC.

The High Performance FICON for System z journey:

First introduced in October of 2008: Initial zHPF announcement with FICON Express4 and FICON Express2 channels

o Maximum of 31k zHPF IO/sec, 2.2 times the FICON protocol

o Single track limit for zHPF data transfers In July 2009 FICON Express8 channels were introduced

o Maximum of 52k zHPF IO/sec o 64k byte limit for zHPF data transfers

In July 2010 additional support was introduced o Extension to multi-tracks of zHPF data transfers

In July 2011 with the new FICON Express8S channel o Introduced a hardware data router for more efficient

zHPF data transfers o FICON Express8S channel became the first channel with

hardware specifically designed to support zHPF. In March 2015 with the new FICON Express16S channel

o FICON Express16S channel became the first channel to support 16Gbps line speed.

Reflected in the bar charts in Figure 1 below are the "best can do" capabilities of each of the last few generations of FICON channels that have been supported on one or more of the following System z servers: z10, z196, zEC12, z13. For each of the recent generations of FICON Express channels, there are two bars, one which displays the maximum capability using the FICON protocol exclusively and another that shows the maximum capability using the zHPF protocol exclusively.

3


4


14,000

31,000

20,000

52,000

23,000

92,00098,000

0

20,000

40,000

60,000

80,000

100,000

120,000

FEx2 & FEx4 FEx2 & FEx4 FEx8 FEx8 FEx8S & FEx16S FEx8S FEx16S

I/O Driver Benchmark maximum IO/sec Small Blocksize

+6.5%

zHPF

zHPF

zHPF

zHPF

350520

620770

620

1,600

2,560

0

500

1,000

1,500

2,000

2,500

3,000

FEx2 & FEx4 FEx2 & FEx4 FEx8 FEx8 FEx8S & FEx16S FEx8S FEx16S

I/O Driver Benchmark maximum MB/sec Large Blocksize

+63%

zHPF

zHPF

zHPF

zHPF

Figure 1

The first chart displays the maximum or 100% channel utilization 4K I/O rates for each channel as measured at a point in time around the general availability (GA) dates of each product using an I/O driver benchmark program for 4K byte read hits. The size of most online database transaction processing (OLTP) workload I/O operations is 4k bytes. In laboratory measurements, using FICON Express16S in a z13

5


with the zHPF protocol and small data transfer I/O operations, FICON Express16S achieved a maximum of 98,000 IOs/sec, compared to the maximum of 92,000 IOs/sec achieved with FICON Express8S. This represents approximately a 6.5% increase. The second chart displays the maximum READ+WRITE MB/sec for each channel.In laboratory measurements, using FICON Express16S in a z13 with the zHPF protocol and a mix of large sequential read and update write data transfer I/O operations, FICON Express16S achieved a maximum throughput of 2560 READ+WRITE MB/sec compared to the maximum of 1600 READ+WRITE MB/sec achieved with FICON Express8S. This represents approximately a 63% increase.

The maximum zHPF 4k IO/sec measured on a FICON Express16S channel was approximately four times the maximum FICON protocol capability. Response time curves for these results will be described later in this document. As shown in Figure 1, with zHPF, it is possible to achieve an improvement in both small block IO/sec processing for OLTP workloads and large sequential READ+WRITE I/O processing compared to previous FICON offerings.

This paper includes the following topics: FICON Express16S channel performance using zHPF and

FICON protocols The High Performance FICON for System z (zHPF) architecture

performance features zHPF exploitation OLTP workload I/O performance examples FICON Express16S Distance Considerations zHPF fields on RMF Channel Activity report FICON Express16S card and IOSS performance with zHPF z13 SAP Capacity

In this paper FICON Express16S may be referred to as FEx16S, FICON Express8S as FEx8S, and FICON Express8 as FEx8. When zHPF is used by itself, it refers to the zHPF protocol. When the term FICON is used by itself, it refers to the FICON protocol. Most of the

6


performance results presented in this paper were obtained using versions of z/OS® available at the time of the measurement. In particular, the FICON Express16S channel measurement results summarized in this paper were collected using z/OS V2.1.

For more information on the performance of earlier generation FICON channels, refer to the material available on the System z I/O connectivity website at:

http://www.ibm.com/systems/z/connectivity/

High Performance FICON for System z (zHPF) architecture performance features

zHPF is an extension to the FICON architecture designed to improve the execution of small block I/O requests. zHPF streamlines the FICON architecture and reduces the overhead on the channel processors, control unit ports, switch ports, and links by improving the way channel programs are written and processed. To understand how zHPF improves upon FICON, one needs to review the relevant characteristics of FICON channel processing.

A FICON channel program consists of a series of Channel Command Words (CCWs) which form a chain. The command code indicates whether the I/O operation is going to be a read or a update write from disk and the count field specifies the number of bytes to transfer. When the channel finishes processing one CCW and either a command chaining or data chaining flag is turned on, it processes the next CCW and the CCWs belonging to such a series are said to be chained. Each one of these CCWs is a FICON channel Information Unit (IU) which requires separate processing on the FICON channel processor and separate commands to be sent across the link from the channel to the control unit. The zHPF architecture defines a single command block to replace a series of FICON CCWs as illustrated in Figure 2 below.

zHPF improves upon FICON by providing a Transport Control Word (TCW) that facilitates the processing of an I/O request by the channel

7



and the control unit. The TCW has a capability that enables multiple channel commands to be sent to the control unit as a single entity instead of being sent as separate commands as is done with FICON CCWs. In addition, the channel is no longer expected to process and keep track of each individual channel command word. Instead, the channel forwards a chain of commands to the control unit to execute. The reduction of this overhead increases the maximum I/O rate possible on the channel and improves the utilization of the various sub-components along the path traversed by the I/O request.

zHPF provides a much simpler link protocol than FICON. Figure 2 below shows an example of a 4k read FICON channel program, where three IUs are sent from the channel to the control unit plus three IUs from the control unit to the channel. In this example, zHPF reduces the total number of IUs sent in half, using one IU from the channel to the control unit and two IUs from the control unit to the channel.

8


CHANNEL

CONTROL

UN IT

OPEN EXCHANGE, PREFIX CMD & DATA

READ COMMAND

CMR

4K of DATA

STATUS

CLOSE EXCHANGE

FICON

Link Protocol Comparison for a 4KB READ

CHANNEL

CON TR O L

UNIT

OPEN EXCHANGE, send a Transport Command IU

4K OF DATA

Send Transport Response IUCLOSE EXCHANGE

zHPF

zHPF provides a much simpler link protocol than FICON

Figure 2

With zHPF, “well constructed” CCW strings are collapsed into a single new Control Word. Conceptually this is similar to the Modified Indirect Data Address Word (MIDAW) facility enhancement to FICON, which allowed a chain of data CCWs to be collapsed into one CCW. zHPF now allows the collapsing of both Command Chained as well as Data Chained CCW strings into one Control Word. zHPF-capable channels and devices support both FICON and zHPF protocols simultaneously.

The maximum number of open exchanges or the number of I/Os that can be simultaneously active on FEx16S channels is designed to be significantly higher with zHPF compared to FICON. An open exchange is an I/O that is active between the channel and the control unit and it includes I/Os that are cache hits, which begin transferring data back to the channel immediately and those that are cache misses which

9


might experience a delay of several milliseconds before the data can begin transferring back to the channel. Since higher I/O activity levels are both possible now and expected to increase in the future with zHPF, the maximum number of open exchanges allowed per channel has been increased with zHPF. More information on how to find the actual average number of open exchanges used in a production workload environment is provided in the “zHPF fields on the RMF Channel Activity report” section of this document.

zHPF ExploitationzHPF can be turned on or off. For z/OS exploitation, there is a parameter in the IECIOSxx member of SYS1.PARMLIB (ZHPF=YES/NO) and on the SETIOS command to control whether zHPF is enabled or disabled. The default is ZHPF=NO.

For zHPF exploitation, FICON Express16S (CHPID type FC) on a z13 requires at a minimum:

z/OS V2.1 z/OS V1.10, V1.11, V1.12 and V1.13 with toleration

support (PTFs)

zHPF-capable channels and devices support both FICON and zHPF protocols simultaneously. The Media Manager component of DFSMS™ detects whether the device supports zHPF or not and builds the appropriate channel programs. Media Manager will build the zHPF Transport Mode channel programs for DB2, PDSE, VSAM, QSAM, BPAM, BSAM, zFS and Extended Format SAM that could benefit from the improved transfer technique.

10


FICON Express16S channel performance using zHPF and FICON protocolsWith the introduction of the FEx16S channel offers performance improvements when using the zHPF protocol to process both small random access and large sequential I/O operations. The hardware data router transfers data both to and from the channel to zEnterprise memory for I/O operations using the zHPF protocol. As a result improvements can be seen in both response times and maximum throughput for IO/sec and MB/sec for workloads using the zHPF protocol on FICON Express16S channels.

One primary performance improvement in a FEx16S channel with zHPF compared to FICON is the maximum number of small block I/Os per second (4k bytes per I/O) that can be processed. As displayed in Figure 3 below, the maximum number of I/Os per second that was measured on a FEx16S channel running an I/O driver benchmark with a 4k bytes per I/O workload exploiting zHPF is approximately 98,000, which is approximately four times what was measured with FICON. In this case, two separate experiments were conducted; one where the channel was executing all zHPF channel programs and another where the channel was executing all FICON channel programs.

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000

Resp

onse

Tim

e (m

sec)

IO/sec

z13 Single Channel FEx16S

FICON zHPF

Figure 3

11


Figure 3 shows that the total response time for a 4k read I/O operation is approximately 60 microseconds better for zHPF compared to FICON channel programs at I/O activity levels less than 10,000 IO/sec per channel. For FICON, response times increase gradually to approximately 0.3ms for activity levels up to 18,000 IO/sec per channel and then increase sharply at a level of activity between 20,000 and 23,000 IO/sec. In contrast, response times for zHPF continue at a moderate rate of increase up to approximately 0.25ms at 90,000 IO/sec, followed by the sharp increase indicative of reaching the maximum capability of the channel. Please note that these measurements were done with a single z13 channel connected to multiple host adapter ports, each on a different IBM System Storage DS8870. Just as is the case with FICON, the maximum capability of any individual control unit (CU) port running zHPF channel programs is different than the maximum capability of the zEnterprise channel. Contact an IBM representative for more information on the performance of a DS8870.

Since the difference in maximum capability with 100% zHPF activity compared to 100% FICON activity is so great on the FEx16S channel, it is worthwhile to note how the maximum capability changes for I/O benchmarks that read 4K bytes of data using a mixture of zHPF and FICON protocols. As shown in the chart below, with a mixture of 90% zHPF and 10% FICON, the maximum capability is 62,000 IO/sec. With a mix of 50% zHPF and 50% FICON, the maximum capability is 33,000 IO/sec on a FEx16S channel. Higher performance improvements can be experienced with higher percent zHPF activity. For information on how to enable zHPF, refer to the “zHPF Exploitation” section of this paper.

12


0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

100,000

100% zHPF 90% zHPF10% FICON

75% zHPF25% FICON

50% zHPF50% FICON

25% zHPF75% FICON

10% zHPF90% FICON

100% FICON

IO/s

ec

FEx16S 4K Read

zHPF FICON

Figure 4

Improvements in maximum megabytes per second (MB/sec) can also be realized with FEx16S channels compared to FEx8S channel as displayed in Figure 5 below.

0

500

1,000

1,500

2,000

2,500

3,000

FEx8S FEx16S

MB/

sec

zHPF Max MB/sec

Read Write Read/Write

+80% +90%

+63%

13


Figure 5

Figure 5 above shows the "best can do" capabilities of both FEx16S and FEx8S channel measured using an I/O driver benchmark program that executes channel programs which use the zHPF protocol to either read multiple sequential 4k records (32x4k = 128k bytes per I/O) of data from disk or write multiple sequential 4k records of data to disk or do a 50/50 mix of read and update write I/O operations. The maximum MB/sec for a single FEx16S channel is approximately 1,500 MB/sec for READs or update WRITEs and 2600 MB/sec for a mix of READ and update WRITE I/O operations. For reference, the maximum MB/sec measured for a single FEx8S channel was approximately 800 MB/sec for READs or update WRITEs and 1,600 MB/sec for a mix of READ and update WRITE I/O operations. Due to the improvement of the 16 Gigabits per second (Gbps) link, the maximum FEx16S channel measurement results are approximately 80% better for all READs and 90% better for all update WRITEs. With a 50/50 mix of READ and update WRITE I/O operations, the maximum capability of the FEx16S channel is 63% greater than the maximum READ+WRITE MB/sec of a single FEx8S channel using the zHPF protocol. In each case using the zHPF protocol, the FEx16S channel is capable of driving the maximum MB/sec that a 16 Gbps link can support.

The following FICON measurements displayed in Figure 6 below are representative of large sequential I/O operations that do not use the zHPF protocol, e.g. those that use EXCP to execute channel programs. These experiments were done using channel programs that transferred six half tracks worth of data or 6x27k bytes for each READ or update WRITE I/O. The maximum MB/sec for a single FEx16S channel using the FICON protocol is approximately 645 MB/sec for READs, 625 MB/sec for update WRITEs and 630 MB/sec for a mix of READ and WRITE I/O operations. The maximum MB/sec for a single FEx8S channel using the FICON protocol is approximately 615 MB/sec for READs, 580 MB/sec for update WRITEs and 620 MB/sec for a mix of READ and WRITE I/O operations.

14


0

100

200

300

400

500

600

700

FEx8S FEx16S

MB/

sec

FICON Max MB/sec


Figure 6

As shown in Figure 7 below, using zHPF on a FEx16S channel, a range from 400 MB/sec for a single 4k block to over 1500 READ MB/sec when 64 or more 4k blocks are transferred per I/O operation was achieved. In contrast, using the FICON protocol with the MIDAW facility, a range from 90 MB/sec for a single 4k block to over 550 READ MB/sec when 64 or more 4k blocks are transferred per I/O operation was measured on a FEx16S channel. The maximum MB/sec possible with zHPF ranges from 3 times FICON for large data transfers (256 k bytes) to 5 times FICON for smaller data transfers (8k bytes per I/O). The graph below Figure 7 demonstrates the performance benefits of the new FEx16S efficiencies of the zHPF protocol, which increases the average bytes per frame and reduces the number of frames per I/O that is transferred.

15


0200400600800

1000120014001600

4 8 16 32 64 128 256 512

MB/

sec

K Bytes of Data Transfer per I/O operation

FEx16S Max Read MB/sec zHPF vs FICON

FICON zHPF

Figure 7

Examples of the response time benefits of both the new FEx16S channel and the efficiencies of the zHPF protocol are shown in Figure 8 below. Simple benchmark experiments were conducted using a FEx16S channel to READ or update WRITE 32 4k records or 128K bytes of data in total to a single device. PEND and CONN time response time components are displayed for zHPF compared to the FICON protocol. For I/O operations that READ 128K bytes of data, zHPF response times are 25% faster than FICON and this is displayed in the CONN time component. For I/O operations that update WRITE 128K bytes of data, zHPF response times are 15% faster than FICON.

16


0.000 0.200 0.400 0.600 0.800 1.000

zHPF Read

FICON Read

zHPF Write

FICON Write

Response Time (msec)

FEx16S Single Stream 32x4K Read

PEND CONN

-15%

-25%

Figure 8

Benefits of both the new FEx16S channel and the efficiencies of the zHPF protocol can also be observed in the channel processor utilization reported on the Resource Measurement Facility (RMF) Channel Activity Report. Figure 9 below shows the FEx16S channel processor utilization reported for both zHPF and FICON measurements for 32x4k READ I/O operations. Since the hardware data router is handling the movement of data for zHPF, the maximum FEx16S channel processor utilization reported for these large data transfer zHPF I/O operations is less than 10%. In contrast the FEx16S channel microprocessor is involved in processing both the Channel Command Words (CCWs) and the transfer of each of the 8k byte data information units (IUs), the channel utilization reported for equivalent amounts of MB/sec processed is more than 20 times what is used for zHPF large data transfers. For large data transfer I/O operations using the zHPF protocol it is therefore more appropriate to observe either channel bus or link utilizations.

17


0

20

40

60

80

100

0 200 400 600 800 1000 1200 1400 1600

Chan

nel U

tiliz

ation

MB/sec

z13 FEx16S Large Data Transfer

FICON zHPF

Figure 9In general, IBM recommends the following guidelines to achieve good response times -- keep channel processor utilizations less than 50% and keep channel link utilizations less than 70%.

When using either the FICON or the zHPF protocol, the FEx16S channel processor utilization is the most important factor to observe for small data transfers. The link or bus utilization is a more important factor to observe for large data transfers using the zHPF protocol. For large data transfers using the FICON protocol, both the channel processor and link utilization guidelines should be observed.

In summary, the FEx16S channel offers performance improvements in both response times and maximum throughput for IO/sec and MB/sec for workloads using the zHPF protocol.

OLTP and Commercial Batch Workload PerformanceThe workload chosen for this experiment is a combination of heritage workloads used to evaluate zEnterprise processor performance. This particular combination consists of a traditional online transaction processing workload (OLTP) made up of moderate to heavy IMS™ transactions from DL/I applications covering diverse business

18


functions, including order entry, stock control, inventory tracking, production specification, hotel reservations, and banking and teller systems along with the CB-L workload that consists of a mix of Commercial Batch Jobs. The batch jobs include various combinations of C, COBOL, FORTRAN, and PL/I compile, link-edit, and execute steps. Sorting, DFSMS utilities (e.g. dump/restore and IEBCOPY), VSAM and DB2 utilities, SQL processing, GDDM® graphics, and FORTRAN engineering/scientific subroutine library processing are also included. The expectation is that this workload would provide ITRR values consistent with the Average Relative Nest Intensity (RNI) column of the Large System Performance Reference (LSPR). https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex?OpenDocument

Experiments were done with this workload to demonstrate the improvements possible in a production environment with FEx8S compared to FEx16S channels.The experiment was performed on an IBM zEnterprise ™ z13 (2964 Model 763) with logical partition (LPAR) defined with 28 Central Processors (CPs) running z/OS2.1. The first experiment used 32 FEx8S channels attached to four IBM System Storage DS8800s (2107-951s) through a FICON Director. The second experiment used 24 FEx16S channels attached to four DS8870s (2107-961s) through a FICON Director.

With an average data transfer size of approximately 8k bytes per I/O and activity level of approximately 241K IO/sec. The z13 with 32 FEx8S channels were 34% busy, for the 24 FEx16S channels the utilizations increased to 35% busy with a 25% reduction in number of channels. In both cases approximately 68% of the I/O operations were converted to use the zHPF protocol.

Improvements in average I/O response times were also observed with these workload experiments as shown in Figure 10 below. From an average of 0.348 ms with FEx8S channels, I/O response times improved by approximately 15% with FEx16S channels.

19

https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex?OpenDocument

https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex?OpenDocument


Figure 10

This experiment demonstrates a range of I/O performance improvements possible with FEx16S channels using a mix of FICON and zHPF protocols. In general, for a variety of workloads, FEx16S channels with zHPF can either allow the number of channels to be reduced while maintaining similar levels of I/O Response Times or with the same number of channels, zHPF can reduce the Channel Utilization or improve the I/O Response Time. FEx16S channels with zHPF may also allow enterprises to increase the number of processors on their zEnterprise servers with no increase in the number of FICON channels.

Every workload is different in terms of the amount of I/O response time and overall transaction response time variation that can be tolerated but connecting two (or more) CU ports to each channel is one example of a reasonable way to consolidate channels.

The FICON Aggregation tool function in zCP3000 can be used to help with channel consolidation and migration from older generation FICON channels to FEx16S channels so that the conversion to zHPF is possible.

The Disk Magic tool is also available to help assess the optimum number of channels needed for a production workload on IBM storage subsystems. Both of these tools use RMF data as input to the analysis process for a specific production environment.

20


FEx16S Channel Performance using zHPF and FICON Protocols at up to 100km of Distance

IBM recommends the use of Cascaded FICON directors with 16Gbps InterSwitch Links (ISLs) between the two directors to support distances up to 100km.

The FEx16S channel card has 90 buffer credits for each individual channel port which is enough to support a 16 Gbps link speed at distances up to 10 km between each channel and its nearest neighbor, which could be a port on a director or tape or disk storage subsystem.

To evaluate FEx16S channel performance using zHPF and FICON protocols at 100 km of distance, the distance was achieved using Dense Wavelength Division Multiplexer (DWDM). A series of performance measurements were conducted with FEx16S channels on z13 connected to multiple storage subsystems (denoted by SS1 and SS2 in the diagram) using cascaded FICON directors with a 16 Gbps InterSwitch Link (ISL) between the two directors extended over a Dense Wavelength Division Multiplexer (DWDM) as illustrated in Figure 11 below.

z13FEx16S

FICONDirector

FICON Director

SS1

16Gbps

SS2100km

8Gbps

8Gbps

16Gbps 16GbpsDWDM

Figure 11

21


The amount of throughput that can be achieved on an ISL at distances up to 100 km is dependent on a number of variables:

Workload characteristics such as data transfer sizes and mix of READ and WRITE activity

zHPF versus FICON protocols Link speed and the bit encoding scheme used on the ISL Buffer credits available on the ISL ports on each of the FICON

directors Actual distance between the two ISL ports on each of the FICON

directors

Figure 12 below displays the maximum MB/sec achieved on a 16 Gbps ISL between two FICON directors separated by 100 km DWDM with both 720 and 960 Buffer-to-Buffer (B2B) credits available on each of the ISL ports. Previously, the general recommendation was to use 0.5 buffers for each 1 km of distance and each 1 Gbps of link speed. This implies 50, 100, 200, 400 and 500 B2B credits for 1, 2, 4, 8 and 10 Gbps link speeds respectively. However, in contrast to the 1, 2, 4 and 8 Gbps FICON link speeds which use 8/10 bit encoding schemes, the 16 Gbps ISL link uses a 64/66 bit encoding scheme. This means that in contrast to the 1, 2, 4 and 8 Gbps FICON channel link speeds which are capable of a maximum of approximately 100, 200, 400 and 800 MB/sec, a 16 Gbps ISL link is capable of a maximum of approximately 1600 MB/sec in each direction. These measurements used 960 B2B credits to ensure the maximum was achieved.

For 128k byte READ or update WRITE I/O operations with 960 B2B credits, a maximum of 1640 MB/sec was achieved using zHPF protocols compared to 1570 MB/sec with FICON protocols. For a 50/50 mix of 128k byte READ and WRITE I/O operations with 960 B2B credits, a maximum of 2990 MB/sec was achieved using zHPF protocols compared to 2690 MB/sec with FICON protocols. For 128k byte READ or update WRITE I/O operations with 720 B2B credits, a maximum of 1470 MB/sec was achieved using zHPF protocols compared to 1350 MB/sec with FICON protocols. For a 50/50 mix of 128k byte READ and WRITE I/O operations with 960 B2B credits, a

22


maximum of 2860 MB/sec was achieved using zHPF protocols compared to 2550 MB/sec with FICON protocols. For the same number of buffer credits, this represents a 4 to 10% improvement in throughput for large data transfers using zHPF protocols compared to FICON protocols. This improvement corresponds to the increase in average bytes per frame using zHPF compared to FICON as shown in Table 1 below. This table also shows the much higher improvements in link efficiency and throughput that is possible with small data transfers of 4k bytes per I/O using zHPF compared to FICON. The increase from 720 to 960 B2B credits yielded an improvement in throughput on the 16 Gbps ISL for both small and large data transfers as shown in Figures 12 and 13.

0

500

1000

1500

2000

2500

3000

3500

100km B2B=720 100km B2B=960

MB/

sec

zHPF 32x4K


+11% +11%

+4%

Figure 12

Table 1 below summarizes the improvement in link efficiency for zHPF compared to FICON for small, medium, and large data transfer sizes. For example, when the frames from a 50/50 mix of 4k byte READ and WRITE I/O operations are flowing across an ISL link, the average bytes per frame using zHPF protocols is 1080 bytes compared to 880 bytes for FICON protocols. This represents a 23% improvement which means that for the same number of IO/sec. For READ I/O operations, the average frame

23


size shown is for the data frames flowing from the ISL port adjacent to the storage subsystem to the ISL port closest to the z13. For update WRITE I/O operations, the average frame size shown is for the data frames flowing to the ISL port adjacent to the storage subsystem from the ISL port closest to the z13.

Data TransferzHPF Average

Frame SizeFICON Average

Frame Size

zHPF improvement

in link efficiency

4K Read or Write 1084 883 23%4K Read/Write 892* (901, 884) 618* (591, 645) 44%

32x4K Read or Write 2025 1938 4%32x4K Read/Write 1981* (1966, 1996) 1800* (1789, 1811) 10%

128x4K Read or Write 2072 1990 4%128x4K Read/Write 2037 * (2010, 2064) 1900* (1900, 1900) 7%

* = average of bytes per frame coming into and going out of ISL port closest to z13

Table 1

0

500

1000

1500

2000

2500

3000

100km B2B=720 100km B2B=960

MB/

sec

FICON 32x4K


+11% +16%

+6%

Figure 13

From a response time perspective, most workloads will experience a 1 millisecond (ms) adder at 100 km of distance compared to local

24


response times due to the speed of light through a fiber. Figure 14 below summarizes this 1ms adder to response times at reasonable levels of activity (approximately 100 MB/sec) for READ I/O operations with small data transfers (4k bytes per I/O), large data transfers (32x4k or 128k bytes per I/O) and very large data transfers (128x4k or approximately 512k bytes per I/O) at 100 km compared to Local distance.

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

FICON4K

FICON32x4K

FICON128x4K

zHPF4K

zHPF32x4K

zHPF128x4K

Resp

onse

Tim

e (m

sec)

Local Distance vs 100km Distance

Local 100km

Figure 14

zHPF Extended Distance IIA new feature introduced on the z13 machine that will allow the channel to control the disabling of first transfer ready before the data is transmitted. This allows the channel to have a high level management of the Control Unit buffers to increase the performance at distance. Extended Distance II (ED II), extends the already available Extended Distance I (ED I) feature for zHPF write operations to support transfers that exceed 64 KB. ED II provides significant latency and overall performance advantage for multiple-track zHPF write operations and is particularly effective as distance between the IBM z13 Server and the IBM DS8870 increases.

25


To evaluate zHPF Extended Distance II, a series of experiments were conducted with FEx16S channel on z13 connected to a DS8870 with 16Gb Host Adapter using a cascaded FICON directors with a 16Gbps Inter Switch Link (ISL) between the two directors extended over a Dense Wavelength Division Multiplexer (DWDM) as illustrated in Figure15 below.

z13FEx16S

FICONDirector

FICON Director SS1

16Gbps

100km

16Gbps16Gbps 16GbpsDWDM

Figure 15

Clients using multi-site configurations can expect up to 66% I/O service time improvement when writing data remotely (remote site recovery). This capability is desired especially for GDPS HyperSwap configurations where the secondary DASD subsystem is in another site.

0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000 8.000 9.000 10.000

ED II zHPF Post-HyperSwap

zHPF Post-HyperSwap

z13 Single Channel FEx16S 128x4K Write Single-Stream

PEND CONN

-66%

26


Figure 16

zHPF fields on the RMF Channel Activity report There are six fields on the RMF Channel Activity report that can be used to distinguish between FICON and zHPF traffic.

The RATE field refers to the number of FICON or zHPF I/Os per second initiated at the total physical channel level (not by LPAR)

The ACTIVE field refers to what we have previously called the "open exchanges", i.e. the number of I/Os that are simultaneously active within a channel.

The DEFER field refers to the number of deferred FICON or zHPF I/O operations per second. This is the number of operations that could not be immediately initiated by the channel due to a temporary lack of resources.

1 C H A N N E L P A T H A C T I V I T Y

PAGE 1z/OS V2R1 SYSTEM ID JL0 DATE 01/15/2015 INTERVAL 09.59.998

RPT VERSION V2R1 RMF TIME 11.00.00 CYCLE 1.000 SECONDS-

IODF = D7 CR-DATE: 01/13/2015 CR-TIME: 09.21.50 ACT: POR MODE: LPAR CPMF: EXTENDED MODE CSSID: 00------------------------------------------------------------------------------------------------------------------------------------

DETAILS FOR ALL CHANNELS-----------------------------------------------------------------------------------------------------------------------------

-------0 CHANNEL PATH UTILIZATION(%) READ(MB/SEC) WRITE(MB/SEC) FICON OPERATIONS ZHPF OPERATIONS

ID TYPE G SPEED SHR PART TOTAL BUS PART TOTAL PART TOTAL RATE ACTIVE DEFER RATE ACTIVE DEFER30 FC_S 17 16G Y 5.34 17.21 2.31 9.53 50.76 3.29 7.08 1483.2 2.9 0.0 205.3 1.1 0.03B FC_S 16 8G Y 1.51 6.32 0.96 2.36 15.50 2.77 8.56 475.5 1.6 0.0 191.8 1.1 0.03D FC_S 15 4G Y 0.13 0.91 0.01 0.02 0.16 0.00 0.01 77.5 1.0 0.0 31.8 1.0 0.0

4B FC_S 13 8G Y 4.74 20.59 4.25 7.68 60.05 3.67 7.89 1343.4 2.6 0.0 196.8 2.2 0.059 FC_S 12 4G Y 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0.0 0.0 0.0 0.0 0.05D FC_S 11 2G Y 0.00 34.90 7.58 0.00 0.00 0.00 121.35 1994.6 4.1 0.0 0.0 0.0 0.0

Figure 17

The example above Figure 17 shows a mix of FICON and zHPF I/O operations on each channel. CHPID 30 executed over 205 zHPF IO/sec in addition to over 1483 FICON IO/sec, the link between the channel and the FICON Director port is operating at a speed of 16Gbps. CHPID 3B executed over 191 zHPF IO/sec in addition to over 475 FICON IO/sec, the link between the channel and the FICON Director port is operating at 8Gbps.

The sample data above shows FEx16S channels operating at different link speeds. zHPF is supported on FEx16S, FEx8S, and FEx8 channels.

27


The FEx16S channels appear with a “15”, “16” or “17” in the generation field of the RMF Channel Activity report. The “15” in the “G” field for CHPID 3D in Figure 16 means the FEx16S channel link has autonegotiated to a speed of 4Gbps. The “16” in the “G” field for CHPID 3B means the FEx16S channel link has autonegotiated to a speed of 8Gbps. The “17” in the “G” field for CHPID 30 means the FEx16S channel link is operating at 16Gbps.

The FEx8S channels appear with a “11”, “12” or “13” in the generation field of the RMF Channel Activity report. A “11” means the FEx8S channel link has autonegotiated to a speed of 2 Gbps. An “12” means the FEx8 channel link has autonegotiated to a speed of 4 Gbps and a “13” means the FEx8S channel link is operating at a speed of 8 Gbps.

FEx8 channels appear with a “7”, “8” or “9” in the generation field of the RMF Channel Activity report. A “7” means the FEx8 channel link has autonegotiated to a speed of 2 Gbps. An “8” means the FEx8 channel link has autonegotiated to a speed of 4 Gbps and a “9” means the FEx8 channel link is operating at a speed of 8 Gbps.

FICON Express16S card and I/O Subsystem (IOSS) performance with zHPF

28


Drawer 3Drawer 1Memory

16GB/s PCIeGen3 x16

Memory Memory Memory

Drawer 0 Drawer 2

PCIeInterconnect

Gen3

OSA-Express5S

PCIe Gen3 PCIe Gen3 ICA SR

PCIeInterconnect

Gen3

PCIeInterconnect

Gen3

FICON Express16S

PCIeInterconnect

Gen3

2 GB/s PCIeGen2 x4

…

PU PU

PU PU PU

PU

SC0

PU PU

PU PU PU

PU PU PU

PU PU PU

PU PU PU

PU PU PU

PU

PCIe I/O drawer

FanoutsFanouts

6 GBps

PCIeSwitch

HCA3-O

RIIRII

8 GB/s PCIe Gen3 x8

12x InfiniBand1x InfiniBand

Coupling Links

4 GB/s PCIeGen2 x8

2 GB/s PCIeGen2 x4

1 GB/s PCIeGen1 x4

2 GB/s PCIeGen1 x8

z13 I/O Infrastructure

SMP Cable

SMP Cable

PCIe Gen3Coupling Links

SC1 SC0 SC1 SC0 SC1 SC0 SC1

SMP Cable

SMP Cable

SMP Cable

SMP Cable

SMP Cable

SMP Cable

Figure 18

The z13 supports the I/O drawer and new form factor I/O cards which use Peripheral Component Interconnect Express Generation 3 (PCIe Gen3) links with increased capacity, granularity, and infrastructure bandwidth, as well as increased reliability, availability, and serviceability. The results of performance measurements done at each of the levels in this new I/O Infrastructure displayed in Figure 18 are summarized in this section.

At the card level, the new FEx16S channel card has two channels on the channel card.

Measurements were done with both FEx16S channels active Figure 19 on the same card to determine the maximum I/Os per second and MB/sec capability of the FEx16S card with zHPF. When each I/O is transferring 4k bytes of data, the maximum IO/sec for two channels on a card is over 191,000 IO/sec or 1.9 times the maximum IO/sec of a single FEx16S channel exploiting the zHPF protocol.

29


When each I/O is transferring 128k bytes of data, the maximum MB/sec for two channels on a single card is approximately 3,000 MB/sec for READs or update WRITEs and over 5,250 MB/sec for a mix or READ and WRITE I/O operations. These card measurement results are about 1.9 to 2.0 times the maximum MB/sec of a single FEx16S channel using the zHPF protocol.

0

50,000

100,000

150,000

200,000

250,000

Channel Card

IO/s

ec

FEx16S max IO/sec


0

1,000

2,000

3,000

4,000

5,000

6,000

Channel Card

IO/s

ec

FEx16S max MB/sec


Figure 19

Figure 20 summarizes the expected I/O bandwidth available at each level of the new z13 I/O Infrastructure including a single FEx16S channel, a single FEx16S card with two FEx16S channels, single PCIe Interconnect domain with up to eight FEx16S channel cards and PCIe fanout card with two PCIe interconnects that allow connections to as many as 16 FEx16S channel cards.

30


0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

Channel Card PCIe interconnect PCIe fanout

MB/

sec

z13 IOSS maximum MB/sec


Figure 20

The results achieved at the channel and card level of the I/O Infrastructure were each described in this document. For the PCIe interconnect level with up to eight FEx16S channel cards, approximately 12 GB/sec for READs or update WRITEs and over 17 GB/sec for a mix or READ and WRITE I/O operations. For the PCIe fanout card with two PCIe Interconnects or up to sixteen FEx16S channel cards, approximately 25 GB/sec for READs or update WRITEs and over 34 GB/sec for a mix or READ and WRITE I/O operations. Which is 18 GB/sec or 120% more than what was measured at the corresponding PCIe Gen2 fanout card level on the zEC12 server.

The z13 supports the Redundant I/O Interconnect feature which is designed to help avoid unplanned outages by maintaining critical connections to I/O devices during an upgrade or repair of a multi-drawer server. The z13 uses an alternate PCIe link which operates at the same speed as the primary PCIe link. The results achieved using two alternate path PCIe links into a single PCIe fanout card were similar to what was achieved using the primary PCIe links, namely 12 READ GB/sec and 12 update WRITE GB/sec.

31


This information about the maximum capabilities of various levels of the z13 IOSS is provided to help customers plan appropriately for high bandwidth demand workloads.

z13 SAP capacity The maximum I/O processing capability on a z13™machine with 24 System Assist Processors (SAPs) is now over 3,200,000 IOs/sec, a significant 32% increase compared to the previous zEC12™ with 16 SAPs.

For z13™, the number of SAPs per processor drawer has been increased from 4 to 6. This growth in IO capacity, increase in number of CPs in a drawer along with the Simultaneous Multi-Threading (SMT) feature for specialty engines leads to a breakthrough capability for IBM customers in terms of IO growth and workload consolidation.

An activity rate of over 3,200,000 Start Subchannels (SSCHs) per second was measured on a z13™ using z/OS V2.1 and a standard configuration of Central Processors (CPs), 24 System Assist Processors (SAPs), and FEx16S channels each executing zHPF small block (4k bytes) I/O operations. The maximum ideal SAP capability for 24 SAPs at 100% SAP utilization is 3.2 Million SSCH/sec. For production environments, IBM recommends that SAP utilizations be kept at or below 70% utilization. Figure 21 below shows I/O capacity measured when running 1, 2, 3 and 4 drawer models of the z13™ and zEC12™ with the SAPs running at 70% utilization levels.

32


1 2 3 40

250000

500000

750000

1000000

1250000

1500000

1750000

2000000

2250000

2500000

z13 vs. zEC12SAP Capacity (70% utilization)

zEC12

z13

Number of Drawers

IOs p

er se

cond

Figure 21

As shown in Figure 21 above:

A z13 Model N30 with 6 SAPs on 1 drawer has an IO capacity of 935,200 SSCHs/sec which is 36% higher than that of zEC12 with 4 SAPs on 1 book.

A z13 Model N63 with 12 SAPs on 2 drawers has an IO capacity of 1,530,200 SSCHs/sec which is 34% higher than that of zEC12 with 8 SAPs on 2 books.

A z13 Model N96 with 18 SAPs on 3 drawers has an IO capacity of 2,068,500 SSCHs/sec which is 36% higher than that of zEC12 with 12 SAPs on 3 books.

A z13 Model NC9 with 24 SAPs on 4 drawers has an IO capacity of 2,262,400 SSCHs/sec which is 32% higher than that of zEC12 with 16 SAPs on 4 books.

33


Summary

The z13 FEx16S channel offers many benefits over previous generations of FICON channels. Improvements can be seen in both response times and maximum IO/sec and MB/sec for workloads using zHPF on FEx16S channels. A single FEx16S channel can support up to either 98,000 zHPF IO/sec for OLTP workloads or 2,600 READ+WRITE MB/sec for high bandwidth applications using the zHPF protocol.

The maximum zHPF 4k IO/sec measured on a FEx16S channel was approximately four times the maximum FICON capability.

zHPF streamlines the FICON architecture and reduces the overhead on the channel processors, control unit ports, switch ports, and links by improving the way channel programs are written and processed. In general, for a variety of workloads, FEx16S channels with zHPF can either allow the number of channels to be reduced while maintaining similar levels of I/O response times or with the same number of channels, zHPF can reduce the channel utilization or improve the I/O response time.

zHPF and FICON rates for FEx16S channels on a z13 are displayed on the RMF Channel Activity report.

The z13 supports the new form factor I/O cards which use PCIe Gen3 links with increased capacity, granularity, and infrastructure bandwidth.

At the card level, the new FEx16S channel card has two channels on a channel card. The z13 FEx16S card capacity is 1.95 to 2.0 times the maximum capability of a single FEx16S channel using the zHPF protocol.

The increased capability of the FEx16S channels are complemented by improved performance at many levels of the z13 I/O Subsystem.

The z13 PCIe fanout card with two PCIe Interconnects or up to sixteen FEx16S channel cards supports up to 34GB/sec Read+Write,

34


120% more than what was measured at the corresponding level on the System zEC12 server.

The maximum I/O processing capability on a z13 with 24 SAPs is over 3,200,000 IO/sec, a 32% increase compared to a zEC12™ with 16 SAPs.

Additional zHPF and FICON product information is available on the zEnterprise I/O connectivity Web site at

http://www.ibm.com/systems/z/connectivity/.

AcknowledgmentsThe data presented in this technical paper is based upon measurements using a mixture of IBM internal tools and non-IBM I/O driver programs, specifically PAI/O™ Driver for z/OS.

The following people contributed to the measurement results presented in this technical paper: Brian Murphy, James Hudspeth, John Morris and Yamil Rivera.

I would also like to thank all of the reviewers of this paper for their helpful comments.

Copyright IBM Corporation 2015IBM CorporationNew Orchard Rd.Armonk, NY 10504U.S.AProduced in the United States of America06/2015All Rights Reserved....

PAI/O is a trademark of Performance Associates, Inc.

35



...

This document contains performance information. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the numbers stated here. The information herein is provided “AS IS” with no warranties, express or implied. This information does not constitute a specification or form part of the warranty for any IBM product.

36

introduction · web view2015/06/26 · ibm z13 and ficon express16s channel performance june 26,...

Documents