flashmemory_mar2014

5/24/2018 FlashMemory_Mar2014

1/32

2013 IBM Corporation

Flash Express

Fernando Ferreira ATS System z - zChampion IBM AcademyMarch 2014Based on zEC12 TLLB


2/32


3/32


Whats New?

Dynamic Reconfiguration of Flash

Express memory

Today, LinuxRHEL V6.4 supportsFlash Express for use withTemporary Storage*

With z/OS V2.1 announcement, CFuse of Flash Express improves resiliency

for WebSphereMQSeriessharedqueues

Support for DB211 use of FlashExpress planned for 2013, rollback ofDB2 10 support via APAR

Java SDK7 SR3 and middleware

exploiters are available

Proof Points

37% increasein transaction throughputduring morning transition

Standalone Dump

19% reduction

in total elapsed time

SVC dump elapsed time

~25% less thanwith DASD

28% improvementin DB2 throughput

WAS Day Trader

8% performance improvement*SLES 11 SP3 expected late 2013

Flash ExpressNotable Increases in Availability and Performance


4/32


What is it? Physically comprised of internal storage stored on Flash SSDs on PCI-e

features

Extends IBMs memory management leadership with a new tier of memory,storage class memory

Provides a secured, resilient and immediately usable solution

Benefits Automatically improves availability for workloads at critical processing times

Drives availability & performance for workloads that cannot tolerate pagingspikes

Slashes latency for application processing such as during diagnosticscollection

Supported use of large pageable (1MB) pages boosts performance

New operations benefit! Eliminating need for PLPA and common pagedatasets

Many planned LE and Middleware exploiters

zEC12/zBC12 offers optional Flash Express memory features Supported in PCIe I/O drawer along with other PCIe I/O features Installed in pairs for high availability

A pair provides 1.4 TB of useable storage

A maximum of 4 pairs ( 4 x 1.4 = 5.6 TB) are supported in a system

No HCD/IOCP definitions required

Storage

Tier

System z

instructions

to read data

Real

Memory(256 B line)

~100

Instructions

Flash

memory (4K

page)

~100 K

instructions

External

Disk (4K

page)

~5000 K

instructions

Latency

Test Results

Higher availability, consistent faster performance, faster start of day processing, less disruptive diagnostics

Flash Express Smarter Availability for Smarter Systems


5/32


What Is Flash Express? Also referred to as Storage Class Memory (SCM)

Flash Express is internal storage implemented via NAND Flash SSDs (Solid State Drives)mounted in PCIe Flash Express feature

Plugs into PCIe I/O drawers in pairs

Data security provided on the feature

A pair provides 1.4 TB of useable storage (1 TB = 230 bytes)

A maximum of 4 pairs ( 4 x 1.4 = 5.6 TB) are supported in a system

Internal Flash Express is accessed using the new System z architected EADM (ExtendedAsynchronous Data Mover) Facility

An extension of the ADM architecture used in the past with expanded storage

Access is initiated with a Start Subchannel instruction

Subchannels used were previously reserved

Definition in IOCDS is not required

The main application of internal Flash Express in zEC12/zBC12 is paging store for z/OS Where it provides advantages in resiliency and speed

With pageable large pages being introduced in tandem for exceptional performance

zEC12 GA2 and zBC12 New Function

CF exploitation of Flash Express memory

All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice. Any relianceon these Statements of General Direction is at the relying party's sole risk and will not create liability or obligation for IBM.


6/32


< 20 ns

5-20 micro sec.

< 10 ms

< 200 ns

Access time

1-3 ms

seconds

CPU

Cache

Random AccessMemory (RAM)

Flash Express

Solid State Drive (SSD) Storage

Spinning Disk Drive

WORM, Tape Library

Relative Access Times for different technologies


7/32


Flash Express

Flash Express exploitation:

z/OS

z/OS V2.1

z/OS V1.13 with PTFs and the z/OS V1.13 RSM Enablement Offering Web deliverable installed

Pagable Large Pages (1 MB)

z/OS V1.13 enabling PTFs for RSM enhancements

Flash Dynamic Reconfiguration

Optional PLPA and COMMON Page data sets

z/OS V1.13 Language Environment

IMS 12 Common Queue Server exploitation with APAR# PM66866

IMS 13 Common Queue Server storage; IMS 13 DB private storage pools*

DB2 V10* planned

Java SDK7 SR3 and by extension exploiters such as

CICS Transaction Server 5.1

WAS Liberty Profile v8.5

IMS 12 available October 2013

DB2 V11

Traditional WAS 8.0.0x and Traditional WAS 8.5.5 (future) **

- Linux on System z distributions:

SLES 11 SP3.

RHEL 6.4.

**Traditional WAS support is planned for a future date*DB2 10 support for V10 with APARs is planned.. IMS 13 support is

planned

.

Flash Express Exploitation on zEC12/zBC12


8/32


Paging

z/OS paging subsystem will work with mix of internal Flash Express and

External Disk Self Tuning based on measured performance

Improved Paging Performance, Simplified Configuration

Begin Paging 1 MB Large Pages only to Flash Express

Exploit Flashs random I/O read rate to gain CPU performance by enabling additionaluse of Large Pages. Currently large pages are not pagable.

Begin Speculative Page-In of 4K Pages

Exploit Flashs random I/O read rate to get Improved Resilience over Disruptions.

Market Open, Workload Failover,

Flash Express is a faster paging device compared to a hard diskThe value is NOT in replacing memory with flash but replacing disk with Flash Express

Flash Express is suitable for workloads that can tolerate paging and will not benefit workloads that cannotafford to page

z/OS FLASH Use Cases


9/32


Pre-fetching on Page Fault

z/OS leverages Flashs fast random reads to bring in additional pages withspatial locality

z/OS is also exploring pre-fetching of more distant pages with temporal localityby employing algorithms such as stride pre-fetching

Compute the strides of the next couple of pages likely to be referenced by recording

the offset of the current target page address and the last two faulting addresses

z/OS Paging Pre-fetching from Flash Express


10/32


LFAREA Parameter

Enhanced IEASYSxx parameter - LFAREA

Enhanced to support the use of 2 GB large pages

The LFAREA parameter can be specified as: LFAREA = ( [1M=req] [,2 GB=req] [,prompt | noprompt] )

Note: The old form of the LFAREA keyword is still supported:

LFAREA = (xx% | mmmmmmM | ggggggG | ttttttT)

Usage Enhancements IARV64 GETSTOR enhanced to support the request for 2 GB large pages


11/32


z/OS Flash Express Use Cases

DumpingMinimize SVC Dump duration, System impact

Flash Express performance during SDUMP

Flash Express performance after SDUMP

Reduce Stand Alone Dump duration

Read time for paged out data


12/32


z/OS Flash Express Use Cases CF

Migration

Structure Real Memory Usage

If migration to Flash keeps up, structure real memorynever fills, so write activity continues to be satisfied

at real memory speedsIf migration does not keep up, there is the potentialthat structure real memory will fill up, at which pointwrites would be delayed until migration catches upenough to free up sufficient real memory objects topermit the new writes to succeed

If prefetching is successful, the GET requests for

objects never have to wait for anything to be broughtback into real, and thus these references continue tobe satisfied at real memory speeds.If prefetching does not keep up, there is the potentialthat an object would be requested/referenced that isstill in Flash. At this point, a Flash fault occurs, andthe request would be delayed until the necessaryobjects are brought back in to real memory.Note: CF requests never wait synchronously for Flashaccesses to take place the delayed requests alwaysreturn to z/OS, who redrives them to the CF in hopesthat the CF will have resolved the Flash fault by then.

Pre Fetch


13/32


+

Flash Express:

Prereqs:

zEC12/zBC12, with Flash Express feature

Flash feature pair memory size is 1.4 TB

Min: 1 feature Pair = > Max: 4 feature

Pairs

Mirroring across pairs for resiliency

Typical configuration: 1 feature pair

Configuration/Setup:

Prior to Enablement; features will be on

line, unformatted

Firmware update delivery will be handled

through zEC12/zBC12 Bundle process

Firmware update via concurrent patch

Config off/on for enablement

Array build via H/W setup (CE/SE)

z/OS (Flash, Pageable Large Pages)

Prereqs:

V1.13 + Web deliverable + SPEs. Webdeliverable also contains enabling PTF:

Dynamic reconfiguration 2GB Pages (not dependent on Flash Express)

4GB Real Storage needed for pageable largepages (not specific to Flash Express)

Configuration/Setup:

At IPL, z/OS detects if flash is assigned to the

partition. z/OS automatically uses FlashExpress for paging unless specified otherwisevia PAGESCM=NONE

All paging data can easily reside on FlashExpress

No need for fine grained capacity planning

No definition required for placement of data on

flash Flash Express uses sub-channels. These are

allocated from the .25K reserved in sub-channel set 0

Flash Express Requirements


14/32


Typical Customer Configurations for FLASH

Flash Express feature pair useable memory size is 1.4TB

Min: 1 feature Pair

Typical customer configuration is 6 to 8 LPARs per CPC and 40 GB -80GB for paging configuration dataset size

Even with 10 LPARs per CPC, each LPAR has 160 GB of flash memoryavailable for its paging datasets, more than double the current typical

customer configuration All paging data can easily reside on Flash Express

Data will preferably go to flash and only go to disk (if any) when flash is full

No intelligent placement of data on internal flash needed


15/32


4 x SSD cards. Each 400 GBs

Flash Express PCIe Adapter feature


16/32


2 ports to formA RAID 10 MirroredPair with the 2nd

Flash Expressfeature

Flash Express PCIe Adapter feature with 2 ports to form aRAID 10 Mirrored Pair


17/32


Twin-Cable interconnect between thetwo Flash Express features in slots 1 and 14

Flash Express - Twin-Cable to Form a RAID 10 Mirrored Pair Flash Express features are always

installed in pairs Maximum 4 pairs in a System

Installed in a PCIe I/O Drawer in 2different I/O Domains

Maximum of 2 pairs installed in adrawer

One Flash Express feature per Domain

only Greater than 2 pairs will require asecond PCIe I/O Drawer

eConfig will reserve a slot in eachDomain in case Flash Express isordered in the future

Features are first installed in the front ofthe installed drawers (slots 1 and 14)before using the rear slots (25 and 33)

Flash Express features are cabledtogether to form a RAID 10 Mirror forredundancy

Data on the Flash feature is protectedwith a unique key stored on the SupportElement (SE) harddisk

Only useable on the system with thekey that encrypted it

Secure Key Store is implemented via aSmart Card that plugs into the SESmart Card reader

Smart Card contains both a unique keypersonalised for each system and asmall Crypto engine that can perform aset of security functions within the card

AES encryption


18/32


z/OS z/OS

System z

MainMemory

SCMSPACE

Data transfer between

Main Memory andStorage Class Memoryis via EADMF (4KB or

1MB blocks)

MainMemory

Storage ClassMemory (SCM)

PartitionMaximum

Partition InitialValuePartition

InitialValue

PartitionInitialValue

PartitionMaximum

LP2LP1

Storage ClassMemory (SCM)

SCMSPACE

Full virtualization of physical

Flash Express features acrosspartitions, software sees an

Abstracted Flash StorageSpace

Allows each logical partition tobe configured with its own SCMaddress space

Allocate Flash Express to

partitions by amount, not cardsize

Ability to change underlyingtechnology while preserving API

No Hardware Specifics in

Software Error Isolation, Transparent

mirroring, Centralizeddiagnostics, etc.

Hardware Logging, FRU Call,Recovery: Independent of

software

Flash Express Virtualization


19/32


Paging Dataset

HDDs or SSDsFLASH

Main Memory

Check Data Characteristics(i.e must reside on flash ormust reside on disk)

If data can reside on either:check space availability

- Flash full If space available on both

check response timestatistics

- Flash is faster

Check Data Characteristics(i.e must reside on flash ormust reside on disk)

If data can reside on either:check space availability

- Flash full If space available on both

check response timestatistics

- Flash is faster

Evict Page

Flash Express vs Disk Placement Criteria


20/32


Data Type Data Page Placement

Pageable Link Pack Area (PLPA) At IPL/NIP time PLPA pages will be placed both on flash and disk.

VIO VIO data will always be placed on disk (First to VIO accepting datasets with any

spillover flowing to nonvio datasets)

HyperSwap Critical AddressSpace data

If flash space is available, all virtual pages belonging to a HyperSwap CriticalAddress Space will be placed on flash memory.If flash space is not available, these pages will be kept in memory and onlypaged to disk when the system is real storage constrained and no otheralternatives exist

Pageable Large Pages If contiguous flash space is available, pageable large pages will bepreferentially written to flash.

All other data If available space exists on both flash and disk then make aselection based on response time.

Flash Express vs Disk Placement Criteria


21/32


Flash Express

Available

Actions

Manage Flash Allocation Task

Select

Add allocation

1

23


22/32


Flash Express Exploitation Considerations

New IEASYSxx parameter - PAGESCM

NONE - do not use flash memory for paging

ALL all flash memory is reserved for paging (default)

0, 0M, 0G, 0T - no flash memory is reserved for paging, allocated as needed fordemand paging

Defaults to ALL

New messages issued at IPL

IAR032I USE OF STORAGE-CLASS MEMORY FOR PAGING IS NOT ENABLED

PAGESCM= NONE

IAR031I USE OF STORAGE-CLASS MEMORY FOR PAGING IS ENABLED -

PAGESCM=ALL, ONLINE=00032768M

IAR031I USE OF STORAGE-CLASS MEMORY FOR PAGING IS ENABLED PAGESCM=00000000M, ONLINE=00032768M

Specification of PLPA and COMMON paging data set is now optional

PLPA/COMMON (*NONE*)


23/32


Flash Express Exploitation Considerations

Display M command

Initial

Maximum - Initial

Maximum


24/32


Flash Express Exploitation - RMF Considerations

RMF Page Data Set Activity (Monitor I and Monitor II):

New entry for SCM space in report and SMF records

RMF Paging Activity (Monitor I): SCM Paging Block data added to report and SMF records

RMF Monitor III STORM Storage Memory Objects Data Table is enhancedto display SCM information


25/32


Integrated Flash memory provides a integrated storage. RAS is provided in layers

The storage is spread over two features which mirror to each other (RAID 10). Should either feature fail, the data

is available on the other feature. Feature replacement is concurrent with customer operations.

The data is encrypted with a volatile key such that it is secure if the feature is removed. Uses smart card on theSupport Element

The data is stored over multiple flash devices in a RAID configuration. Should a flash device fail the data is

reconstructed dynamically.

Firmware recovery for general feature failures

Firmware updates are concurrent with customer operations.

Addressing on the PCIe bus to the host is protected by a SAP firmware protocol

The PCIe bus benefits from the redundant I/O interconnect.

Integrated Flash Express

Layered RAS of integrated flash memory

PCIe

PCIe

PCIe IO Drawer

Flash Express Card

PCIe

RAIDController

Flash Express Card

PCIe

RAIDController

PCIe

Switch

PCIe

SwitchUniquePROTOCOL

UniquePROTOCOL

SSD

SSD

SSD

SSD

SupportElement

SupportElement

ECC Protection& Encryption

SSD

SSD

SSD

SSD

ECC Protection

& Encryption


26/32


Security of Data on Flash Express

System z internal flash can be used for paging, dumping and .....

It can contain all data, including audited personally identifiable data

Client data on flash is protected by AES Encryption

Done using hardware encryption at the device like IBMs Disk and Tape encryption

Key Management is provided based on a Smart Card Smart Card Reader installed in the Support Element

End of life Audit is based on access to the Smart Card, not access to the Flash Memory

Secure Cryptographic Erase well understood

SE HD

SE

IOP

HSA RSA Key Pair storage

PrivatePublic

Flash

SE

ImbeddedKey

Controller

GET_IOP_PUBLIC

SET_IOP_DEVKEY

RSA Key Pair storage

Private


PrivatePublic


Private


27/32


Integrated Key Controller detail

Support Element (SE)

SE

Hard DiskIntegratedKey

Controller

SupportElement

Serial NumberFlash Encryption Key /

Authentication Key

RSA Private Key

Keys Generated in the Smart Card

AES Flash Encryption Key /Authentication Key

AES Key-Encrypting Key

RSA Public Key


28/32


Support Element (SE)

Integrated

KeyController

RSA Public Key

Keys Generated in the Smart Card

AES Flash Encryption Key /Authentication Key

RSA Public Key

AES Key-Encrypting Key

Firmware RSA Key Pair HSA

Firmware Management of the FlashExpress Adapter

Flash Encryption Key /Authentication Key

Flash

Public Key Private Key

SE

Hard Disk

Firmware Public Key

AES FlashEncryption Key /

Authentication Key

Private Key

Flash Encryption Key /Authentication Key

AES FlashEncryption Key /

Authentication Key

Key Serving Topology detail


29/32


1. Workload Transition During morning transition, workloads using Flash Express reached peak throughput in under 1/4th the timePaging to Flash Express during morning transition showed up to a 10 times faster response time and up to a 37% increase in

throughput in the first 45 secondsVALUE: Ability to start up call center, start trading, resume transactions during heavy morning open period without delays

2. WAS Java Performance Benchmarks The WAS Day Trader benchmarks showed up to an 8% performance improvement using Flash ExpressVALUE: Performance improvements for workloads that leverage Javas use of pageable large pagesThis test used 64-bit Java 7 SR3 with JIT code cache & Java Heap leveraging Flash and pageable large pages.

3. Diagnostics In SVC dumps, availability was up to 4 times higher for workloads and up to twice as high for systems; steady state performance was

achieved up to 4 times faster In Standalone dump test, a 36 GB dump with Flash Express had a 19% reduction in total dump timeTransaction steady state was reached in 14 seconds with Flash Express, vs. 60 seconds with DASD.VALUE: During SVC and Standalone dumps, System and Address spaces availability was dramatically increased. This enabled work to

be processed that would otherwise have been stopped.

4. DB2 Up to 28% improvement in DB2 throughput due to faster CPU plus Flash Express with Pageable Large Pages*Workloads leveraging Flash Express with PLP can see up to a 8%** price performance improvement over the z196.VALUE: Ability to realize price performance value and faster throughput

* PLP for DB2 helps DB2 to achieve up to 3% additional performance on top of zEC12 CPU expected throughput improvements of25%.

** Based on average 5% discount for zEC12 workloads under the AWLC pricing plus up to 3% more performance per MSU with FlashExpress.

Note All tests are comparing the use of Flash Express as compared to using DASD (DS8800)

Proof Points Summary for Flash Express


30/32


31/32


Additional Information

z/OS Home Page

http://www.ibm.com/servers/eserver/zseries/z/OS/

zFavorites for System z

http://www.ibm.com/servers/eserver/zseries/z/OS/zfavorites/

z/OS Internet Libraryhttp://www.ibm.com/servers/eserver/zseries/z/OS/bkserv/

IBM System z

http://www.ibm.com/systems/z/

IBM Resource Link

https://app-06.www.ibm.com/servers/resourcelink/hom03010.nsf

IBM Redbooks How-To Books (also Redpieces)

http://www.redbooks.ibm.com/

Preventive Service Planning buckets

http://www14.software.ibm.com/webapp/set2/psp/srchBroker Enhanced HOLDDATA

http://service.boulder.ibm.com/390holddata.html


32/32


ZSP03588-USEN-02

flashmemory_mar2014

Documents