Download - FlashMemory_Mar2014
-
5/24/2018 FlashMemory_Mar2014
1/32
2013 IBM Corporation
Flash Express
Fernando Ferreira ATS System z - zChampion IBM AcademyMarch 2014Based on zEC12 TLLB
-
5/24/2018 FlashMemory_Mar2014
2/32
-
5/24/2018 FlashMemory_Mar2014
3/32
2013 IBM Corporation
Whats New?
Dynamic Reconfiguration of Flash
Express memory
Today, LinuxRHEL V6.4 supportsFlash Express for use withTemporary Storage*
With z/OS V2.1 announcement, CFuse of Flash Express improves resiliency
for WebSphereMQSeriessharedqueues
Support for DB211 use of FlashExpress planned for 2013, rollback ofDB2 10 support via APAR
Java SDK7 SR3 and middleware
exploiters are available
Proof Points
37% increasein transaction throughputduring morning transition
Standalone Dump
19% reduction
in total elapsed time
SVC dump elapsed time
~25% less thanwith DASD
28% improvementin DB2 throughput
WAS Day Trader
8% performance improvement*SLES 11 SP3 expected late 2013
Flash ExpressNotable Increases in Availability and Performance
-
5/24/2018 FlashMemory_Mar2014
4/32
2013 IBM Corporation
What is it? Physically comprised of internal storage stored on Flash SSDs on PCI-e
features
Extends IBMs memory management leadership with a new tier of memory,storage class memory
Provides a secured, resilient and immediately usable solution
Benefits Automatically improves availability for workloads at critical processing times
Drives availability & performance for workloads that cannot tolerate pagingspikes
Slashes latency for application processing such as during diagnosticscollection
Supported use of large pageable (1MB) pages boosts performance
New operations benefit! Eliminating need for PLPA and common pagedatasets
Many planned LE and Middleware exploiters
zEC12/zBC12 offers optional Flash Express memory features Supported in PCIe I/O drawer along with other PCIe I/O features Installed in pairs for high availability
A pair provides 1.4 TB of useable storage
A maximum of 4 pairs ( 4 x 1.4 = 5.6 TB) are supported in a system
No HCD/IOCP definitions required
Storage
Tier
System z
instructions
to read data
Real
Memory(256 B line)
~100
Instructions
Flash
memory (4K
page)
~100 K
instructions
External
Disk (4K
page)
~5000 K
instructions
Latency
Test Results
Higher availability, consistent faster performance, faster start of day processing, less disruptive diagnostics
Flash Express Smarter Availability for Smarter Systems
-
5/24/2018 FlashMemory_Mar2014
5/32
2013 IBM Corporation
What Is Flash Express? Also referred to as Storage Class Memory (SCM)
Flash Express is internal storage implemented via NAND Flash SSDs (Solid State Drives)mounted in PCIe Flash Express feature
Plugs into PCIe I/O drawers in pairs
Data security provided on the feature
A pair provides 1.4 TB of useable storage (1 TB = 230 bytes)
A maximum of 4 pairs ( 4 x 1.4 = 5.6 TB) are supported in a system
Internal Flash Express is accessed using the new System z architected EADM (ExtendedAsynchronous Data Mover) Facility
An extension of the ADM architecture used in the past with expanded storage
Access is initiated with a Start Subchannel instruction
Subchannels used were previously reserved
Definition in IOCDS is not required
The main application of internal Flash Express in zEC12/zBC12 is paging store for z/OS Where it provides advantages in resiliency and speed
With pageable large pages being introduced in tandem for exceptional performance
zEC12 GA2 and zBC12 New Function
CF exploitation of Flash Express memory
All statements regarding IBM's plans, directions, and intent are subject to change or withdrawal without notice. Any relianceon these Statements of General Direction is at the relying party's sole risk and will not create liability or obligation for IBM.
-
5/24/2018 FlashMemory_Mar2014
6/32
2013 IBM Corporation
< 20 ns
5-20 micro sec.
< 10 ms
< 200 ns
Access time
1-3 ms
seconds
CPU
Cache
Random AccessMemory (RAM)
Flash Express
Solid State Drive (SSD) Storage
Spinning Disk Drive
WORM, Tape Library
Relative Access Times for different technologies
-
5/24/2018 FlashMemory_Mar2014
7/32
2013 IBM Corporation
Flash Express
Flash Express exploitation:
z/OS
z/OS V2.1
z/OS V1.13 with PTFs and the z/OS V1.13 RSM Enablement Offering Web deliverable installed
Pagable Large Pages (1 MB)
z/OS V1.13 enabling PTFs for RSM enhancements
Flash Dynamic Reconfiguration
Optional PLPA and COMMON Page data sets
z/OS V1.13 Language Environment
IMS 12 Common Queue Server exploitation with APAR# PM66866
IMS 13 Common Queue Server storage; IMS 13 DB private storage pools*
DB2 V10* planned
Java SDK7 SR3 and by extension exploiters such as
CICS Transaction Server 5.1
WAS Liberty Profile v8.5
IMS 12 available October 2013
DB2 V11
Traditional WAS 8.0.0x and Traditional WAS 8.5.5 (future) **
- Linux on System z distributions:
SLES 11 SP3.
RHEL 6.4.
**Traditional WAS support is planned for a future date*DB2 10 support for V10 with APARs is planned.. IMS 13 support is
planned
.
Flash Express Exploitation on zEC12/zBC12
-
5/24/2018 FlashMemory_Mar2014
8/32
2013 IBM Corporation
Paging
z/OS paging subsystem will work with mix of internal Flash Express and
External Disk Self Tuning based on measured performance
Improved Paging Performance, Simplified Configuration
Begin Paging 1 MB Large Pages only to Flash Express
Exploit Flashs random I/O read rate to gain CPU performance by enabling additionaluse of Large Pages. Currently large pages are not pagable.
Begin Speculative Page-In of 4K Pages
Exploit Flashs random I/O read rate to get Improved Resilience over Disruptions.
Market Open, Workload Failover,
Flash Express is a faster paging device compared to a hard diskThe value is NOT in replacing memory with flash but replacing disk with Flash Express
Flash Express is suitable for workloads that can tolerate paging and will not benefit workloads that cannotafford to page
z/OS FLASH Use Cases
-
5/24/2018 FlashMemory_Mar2014
9/32
2013 IBM Corporation
Pre-fetching on Page Fault
z/OS leverages Flashs fast random reads to bring in additional pages withspatial locality
z/OS is also exploring pre-fetching of more distant pages with temporal localityby employing algorithms such as stride pre-fetching
Compute the strides of the next couple of pages likely to be referenced by recording
the offset of the current target page address and the last two faulting addresses
z/OS Paging Pre-fetching from Flash Express
-
5/24/2018 FlashMemory_Mar2014
10/32
2013 IBM Corporation
LFAREA Parameter
Enhanced IEASYSxx parameter - LFAREA
Enhanced to support the use of 2 GB large pages
The LFAREA parameter can be specified as: LFAREA = ( [1M=req] [,2 GB=req] [,prompt | noprompt] )
Note: The old form of the LFAREA keyword is still supported:
LFAREA = (xx% | mmmmmmM | ggggggG | ttttttT)
Usage Enhancements IARV64 GETSTOR enhanced to support the request for 2 GB large pages
-
5/24/2018 FlashMemory_Mar2014
11/32
2013 IBM Corporation
z/OS Flash Express Use Cases
DumpingMinimize SVC Dump duration, System impact
Flash Express performance during SDUMP
Flash Express performance after SDUMP
Reduce Stand Alone Dump duration
Read time for paged out data
-
5/24/2018 FlashMemory_Mar2014
12/32
2013 IBM Corporation
z/OS Flash Express Use Cases CF
Migration
Structure Real Memory Usage
If migration to Flash keeps up, structure real memorynever fills, so write activity continues to be satisfied
at real memory speedsIf migration does not keep up, there is the potentialthat structure real memory will fill up, at which pointwrites would be delayed until migration catches upenough to free up sufficient real memory objects topermit the new writes to succeed
If prefetching is successful, the GET requests for
objects never have to wait for anything to be broughtback into real, and thus these references continue tobe satisfied at real memory speeds.If prefetching does not keep up, there is the potentialthat an object would be requested/referenced that isstill in Flash. At this point, a Flash fault occurs, andthe request would be delayed until the necessaryobjects are brought back in to real memory.Note: CF requests never wait synchronously for Flashaccesses to take place the delayed requests alwaysreturn to z/OS, who redrives them to the CF in hopesthat the CF will have resolved the Flash fault by then.
Pre Fetch
-
5/24/2018 FlashMemory_Mar2014
13/32
2013 IBM Corporation
+
Flash Express:
Prereqs:
zEC12/zBC12, with Flash Express feature
Flash feature pair memory size is 1.4 TB
Min: 1 feature Pair = > Max: 4 feature
Pairs
Mirroring across pairs for resiliency
Typical configuration: 1 feature pair
Configuration/Setup:
Prior to Enablement; features will be on
line, unformatted
Firmware update delivery will be handled
through zEC12/zBC12 Bundle process
Firmware update via concurrent patch
Config off/on for enablement
Array build via H/W setup (CE/SE)
z/OS (Flash, Pageable Large Pages)
Prereqs:
V1.13 + Web deliverable + SPEs. Webdeliverable also contains enabling PTF:
Dynamic reconfiguration 2GB Pages (not dependent on Flash Express)
4GB Real Storage needed for pageable largepages (not specific to Flash Express)
Configuration/Setup:
At IPL, z/OS detects if flash is assigned to the
partition. z/OS automatically uses FlashExpress for paging unless specified otherwisevia PAGESCM=NONE
All paging data can easily reside on FlashExpress
No need for fine grained capacity planning
No definition required for placement of data on
flash Flash Express uses sub-channels. These are
allocated from the .25K reserved in sub-channel set 0
Flash Express Requirements
-
5/24/2018 FlashMemory_Mar2014
14/32
2013 IBM Corporation
Typical Customer Configurations for FLASH
Flash Express feature pair useable memory size is 1.4TB
Min: 1 feature Pair
Typical customer configuration is 6 to 8 LPARs per CPC and 40 GB -80GB for paging configuration dataset size
Even with 10 LPARs per CPC, each LPAR has 160 GB of flash memoryavailable for its paging datasets, more than double the current typical
customer configuration All paging data can easily reside on Flash Express
Data will preferably go to flash and only go to disk (if any) when flash is full
No intelligent placement of data on internal flash needed
-
5/24/2018 FlashMemory_Mar2014
15/32
2013 IBM Corporation
4 x SSD cards. Each 400 GBs
Flash Express PCIe Adapter feature
-
5/24/2018 FlashMemory_Mar2014
16/32
2013 IBM Corporation
2 ports to formA RAID 10 MirroredPair with the 2nd
Flash Expressfeature
Flash Express PCIe Adapter feature with 2 ports to form aRAID 10 Mirrored Pair
-
5/24/2018 FlashMemory_Mar2014
17/32
2013 IBM Corporation
Twin-Cable interconnect between thetwo Flash Express features in slots 1 and 14
Flash Express - Twin-Cable to Form a RAID 10 Mirrored Pair Flash Express features are always
installed in pairs Maximum 4 pairs in a System
Installed in a PCIe I/O Drawer in 2different I/O Domains
Maximum of 2 pairs installed in adrawer
One Flash Express feature per Domain
only Greater than 2 pairs will require asecond PCIe I/O Drawer
eConfig will reserve a slot in eachDomain in case Flash Express isordered in the future
Features are first installed in the front ofthe installed drawers (slots 1 and 14)before using the rear slots (25 and 33)
Flash Express features are cabledtogether to form a RAID 10 Mirror forredundancy
Data on the Flash feature is protectedwith a unique key stored on the SupportElement (SE) harddisk
Only useable on the system with thekey that encrypted it
Secure Key Store is implemented via aSmart Card that plugs into the SESmart Card reader
Smart Card contains both a unique keypersonalised for each system and asmall Crypto engine that can perform aset of security functions within the card
AES encryption
-
5/24/2018 FlashMemory_Mar2014
18/32
2013 IBM Corporation
z/OS z/OS
System z
MainMemory
SCMSPACE
Data transfer between
Main Memory andStorage Class Memoryis via EADMF (4KB or
1MB blocks)
MainMemory
Storage ClassMemory (SCM)
PartitionMaximum
Partition InitialValuePartition
InitialValue
PartitionInitialValue
PartitionMaximum
LP2LP1
Storage ClassMemory (SCM)
SCMSPACE
Full virtualization of physical
Flash Express features acrosspartitions, software sees an
Abstracted Flash StorageSpace
Allows each logical partition tobe configured with its own SCMaddress space
Allocate Flash Express to
partitions by amount, not cardsize
Ability to change underlyingtechnology while preserving API
No Hardware Specifics in
Software Error Isolation, Transparent
mirroring, Centralizeddiagnostics, etc.
Hardware Logging, FRU Call,Recovery: Independent of
software
Flash Express Virtualization
-
5/24/2018 FlashMemory_Mar2014
19/32
2013 IBM Corporation
Paging Dataset
HDDs or SSDsFLASH
Main Memory
Check Data Characteristics(i.e must reside on flash ormust reside on disk)
If data can reside on either:check space availability
- Flash full If space available on both
check response timestatistics
- Flash is faster
Check Data Characteristics(i.e must reside on flash ormust reside on disk)
If data can reside on either:check space availability
- Flash full If space available on both
check response timestatistics
- Flash is faster
Evict Page
Flash Express vs Disk Placement Criteria
-
5/24/2018 FlashMemory_Mar2014
20/32
2013 IBM Corporation
Data Type Data Page Placement
Pageable Link Pack Area (PLPA) At IPL/NIP time PLPA pages will be placed both on flash and disk.
VIO VIO data will always be placed on disk (First to VIO accepting datasets with any
spillover flowing to nonvio datasets)
HyperSwap Critical AddressSpace data
If flash space is available, all virtual pages belonging to a HyperSwap CriticalAddress Space will be placed on flash memory.If flash space is not available, these pages will be kept in memory and onlypaged to disk when the system is real storage constrained and no otheralternatives exist
Pageable Large Pages If contiguous flash space is available, pageable large pages will bepreferentially written to flash.
All other data If available space exists on both flash and disk then make aselection based on response time.
Flash Express vs Disk Placement Criteria
-
5/24/2018 FlashMemory_Mar2014
21/32
2013 IBM Corporation
Flash Express
Available
Actions
Manage Flash Allocation Task
Select
Add allocation
1
23
-
5/24/2018 FlashMemory_Mar2014
22/32
2013 IBM Corporation
Flash Express Exploitation Considerations
New IEASYSxx parameter - PAGESCM
NONE - do not use flash memory for paging
ALL all flash memory is reserved for paging (default)
0, 0M, 0G, 0T - no flash memory is reserved for paging, allocated as needed fordemand paging
Defaults to ALL
New messages issued at IPL
IAR032I USE OF STORAGE-CLASS MEMORY FOR PAGING IS NOT ENABLED
PAGESCM= NONE
IAR031I USE OF STORAGE-CLASS MEMORY FOR PAGING IS ENABLED -
PAGESCM=ALL, ONLINE=00032768M
IAR031I USE OF STORAGE-CLASS MEMORY FOR PAGING IS ENABLED PAGESCM=00000000M, ONLINE=00032768M
Specification of PLPA and COMMON paging data set is now optional
PLPA/COMMON (*NONE*)
-
5/24/2018 FlashMemory_Mar2014
23/32
2013 IBM Corporation
Flash Express Exploitation Considerations
Display M command
Initial
Maximum - Initial
Maximum
-
5/24/2018 FlashMemory_Mar2014
24/32
2013 IBM Corporation
Flash Express Exploitation - RMF Considerations
RMF Page Data Set Activity (Monitor I and Monitor II):
New entry for SCM space in report and SMF records
RMF Paging Activity (Monitor I): SCM Paging Block data added to report and SMF records
RMF Monitor III STORM Storage Memory Objects Data Table is enhancedto display SCM information
-
5/24/2018 FlashMemory_Mar2014
25/32
2013 IBM Corporation
Integrated Flash memory provides a integrated storage. RAS is provided in layers
The storage is spread over two features which mirror to each other (RAID 10). Should either feature fail, the data
is available on the other feature. Feature replacement is concurrent with customer operations.
The data is encrypted with a volatile key such that it is secure if the feature is removed. Uses smart card on theSupport Element
The data is stored over multiple flash devices in a RAID configuration. Should a flash device fail the data is
reconstructed dynamically.
Firmware recovery for general feature failures
Firmware updates are concurrent with customer operations.
Addressing on the PCIe bus to the host is protected by a SAP firmware protocol
The PCIe bus benefits from the redundant I/O interconnect.
Integrated Flash Express
Layered RAS of integrated flash memory
PCIe
PCIe
PCIe IO Drawer
Flash Express Card
PCIe
RAIDController
Flash Express Card
PCIe
RAIDController
PCIe
Switch
PCIe
SwitchUniquePROTOCOL
UniquePROTOCOL
SSD
SSD
SSD
SSD
SupportElement
SupportElement
ECC Protection& Encryption
SSD
SSD
SSD
SSD
ECC Protection
& Encryption
-
5/24/2018 FlashMemory_Mar2014
26/32
2013 IBM Corporation
Security of Data on Flash Express
System z internal flash can be used for paging, dumping and .....
It can contain all data, including audited personally identifiable data
Client data on flash is protected by AES Encryption
Done using hardware encryption at the device like IBMs Disk and Tape encryption
Key Management is provided based on a Smart Card Smart Card Reader installed in the Support Element
End of life Audit is based on access to the Smart Card, not access to the Flash Memory
Secure Cryptographic Erase well understood
SE HD
SE
IOP
HSA RSA Key Pair storage
PrivatePublic
Flash
SE
ImbeddedKey
Controller
GET_IOP_PUBLIC
SET_IOP_DEVKEY
RSA Key Pair storage
Private
HSA RSA Key Pair storage
PrivatePublic
HSA RSA Key Pair storage
Private
-
5/24/2018 FlashMemory_Mar2014
27/32
2013 IBM Corporation
Integrated Key Controller detail
Support Element (SE)
SE
Hard DiskIntegratedKey
Controller
SupportElement
Serial NumberFlash Encryption Key /
Authentication Key
RSA Private Key
Keys Generated in the Smart Card
AES Flash Encryption Key /Authentication Key
AES Key-Encrypting Key
RSA Public Key
-
5/24/2018 FlashMemory_Mar2014
28/32
2013 IBM Corporation
Support Element (SE)
Integrated
KeyController
RSA Public Key
Keys Generated in the Smart Card
AES Flash Encryption Key /Authentication Key
RSA Public Key
AES Key-Encrypting Key
Firmware RSA Key Pair HSA
Firmware Management of the FlashExpress Adapter
Flash Encryption Key /Authentication Key
Flash
Public Key Private Key
SE
Hard Disk
Firmware Public Key
AES FlashEncryption Key /
Authentication Key
Private Key
Flash Encryption Key /Authentication Key
AES FlashEncryption Key /
Authentication Key
Key Serving Topology detail
-
5/24/2018 FlashMemory_Mar2014
29/32
2013 IBM Corporation
1. Workload Transition During morning transition, workloads using Flash Express reached peak throughput in under 1/4th the timePaging to Flash Express during morning transition showed up to a 10 times faster response time and up to a 37% increase in
throughput in the first 45 secondsVALUE: Ability to start up call center, start trading, resume transactions during heavy morning open period without delays
2. WAS Java Performance Benchmarks The WAS Day Trader benchmarks showed up to an 8% performance improvement using Flash ExpressVALUE: Performance improvements for workloads that leverage Javas use of pageable large pagesThis test used 64-bit Java 7 SR3 with JIT code cache & Java Heap leveraging Flash and pageable large pages.
3. Diagnostics In SVC dumps, availability was up to 4 times higher for workloads and up to twice as high for systems; steady state performance was
achieved up to 4 times faster In Standalone dump test, a 36 GB dump with Flash Express had a 19% reduction in total dump timeTransaction steady state was reached in 14 seconds with Flash Express, vs. 60 seconds with DASD.VALUE: During SVC and Standalone dumps, System and Address spaces availability was dramatically increased. This enabled work to
be processed that would otherwise have been stopped.
4. DB2 Up to 28% improvement in DB2 throughput due to faster CPU plus Flash Express with Pageable Large Pages*Workloads leveraging Flash Express with PLP can see up to a 8%** price performance improvement over the z196.VALUE: Ability to realize price performance value and faster throughput
* PLP for DB2 helps DB2 to achieve up to 3% additional performance on top of zEC12 CPU expected throughput improvements of25%.
** Based on average 5% discount for zEC12 workloads under the AWLC pricing plus up to 3% more performance per MSU with FlashExpress.
Note All tests are comparing the use of Flash Express as compared to using DASD (DS8800)
Proof Points Summary for Flash Express
-
5/24/2018 FlashMemory_Mar2014
30/32
-
5/24/2018 FlashMemory_Mar2014
31/32
2013 IBM Corporation
Additional Information
z/OS Home Page
http://www.ibm.com/servers/eserver/zseries/z/OS/
zFavorites for System z
http://www.ibm.com/servers/eserver/zseries/z/OS/zfavorites/
z/OS Internet Libraryhttp://www.ibm.com/servers/eserver/zseries/z/OS/bkserv/
IBM System z
http://www.ibm.com/systems/z/
IBM Resource Link
https://app-06.www.ibm.com/servers/resourcelink/hom03010.nsf
IBM Redbooks How-To Books (also Redpieces)
http://www.redbooks.ibm.com/
Preventive Service Planning buckets
http://www14.software.ibm.com/webapp/set2/psp/srchBroker Enhanced HOLDDATA
http://service.boulder.ibm.com/390holddata.html
-
5/24/2018 FlashMemory_Mar2014
32/32
2013 IBM Corporation
ZSP03588-USEN-02