Draft Document for Review February 7, 2014 7:33 pm SG24-8189-00

ibm.com/redbooks

IBM Front cover

Implementing IBM FlashSystem 840

Karen OrlandoMarcos Figueirdo Jr.

Adrian FloresDetlef Helmbrecht

Ron HerrmannCarsten Larsen

Leandro Torolho

Exteme performance with IBM MicroLatency and Macro efficiency

Improve system availability with Concurrent code load

New intuitive Graphical User Interface

International Technical Support Organization

Implementing IBM FlashSystem 840 Storage

February 2014

Draft Document for Review February 7, 2014 7:33 pm 8189edno.fm

SG24-8189-00

8189edno.fm Draft Document for Review February 7, 2014 7:33 pmFirst Edition (February 2014)This edition applies to IBM FlashSystem 840.

This document was created or updated on February 7, 2014.

Note: Before using this information and the product it supports, read the information in Notices on page xvii. Copyright International Business Machines Corporation 2013. All rights reserved.Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP ScheduleContract with IBM Corp.

Draft Document for Review February 7, 2014 7:33 pm 8189edno.fm iii

8189edno.fm Draft Document for Review February 7, 2014 7:33 pmiv Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189TOC.fmContents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiAuthors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivStay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii

Chapter 1. FlashSystem Storage Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 FlashSystem Storage Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.2 Why Flash matters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 IBM FlashSystem family: Product Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Technology and Architectural Design Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4.1 Variable Stripe RAID and two dimensional Flash RAID Overview . . . . . . . . . . . . . 71.5 Variable Stripe RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.6 How VSR works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91.7 Two dimensional (2D) Flash RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Chapter 2. IBM FlashSystem 840 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.1 Introduction to IBM FlashSystem 840 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.1.1 IBM FlashSystem 840 capacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.1.2 IBM FlashSystem 840 performance and latency . . . . . . . . . . . . . . . . . . . . . . . . . 152.1.3 IBM FlashSystem 840 power requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.1.4 IBM FlashSystem 840 physical specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.1.5 IBM FlashSystem 840 reliability and serviceability . . . . . . . . . . . . . . . . . . . . . . . . 172.1.6 IBM FlashSystem 840 scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.1.7 IBM FlashSystem 840 protocol support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.1.8 IBM FlashSystem 840 encryption support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.1.9 IBM FlashSystem Models 820 and 840 comparison. . . . . . . . . . . . . . . . . . . . . . . 202.1.10 IBM FlashSystem 840 Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.2 IBM FlashSystem 840 architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.2.1 IBM FlashSystem 840 architecture overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.2.2 IBM FlashSystem 840 hardware components . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.2.3 IBM FlashSystem 840 Canisters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.2.4 IBM FlashSystem 840 interface cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252.2.5 IBM FlashSystem 840 flash modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282.2.6 IBM FlashSystem 840 battery modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.3 IBM FlashSystem 840 administration and maintenance . . . . . . . . . . . . . . . . . . . . . . . . 332.3.1 IBM FlashSystem 840 serviceability and software enhancements . . . . . . . . . . . . 332.3.2 IBM FlashSystem 840 system management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.4 IBM FlashSystem 840 support matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392.5 IBM FlashSystem 840 IBM product integration overview . . . . . . . . . . . . . . . . . . . . . . . 40

2.5.1 IBM SAN Volume Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.5.2 IBM Storwize V7000 storage array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.5.3 IBM PureFlex and PureSystems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.5.4 IBM DB2 database environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.5.5 IBM General Parallel File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Chapter 3. Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Copyright IBM Corp. 2013. All rights reserved. v

8189TOC.fm Draft Document for Review February 7, 2014 7:33 pm3.1 Installation pre-requisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.1.1 Completing the hardware location chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.1.2 Planning for power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

3.2 System management (IP addresses, supported browser) . . . . . . . . . . . . . . . . . . . . . . 483.2.1 Management IP address. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.2.2 Service IP addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.2.3 Additional management addresses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.2.4 Checking your web browser settings for the management GUI . . . . . . . . . . . . . . 50

3.3 Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523.4 Supported hosts and operating system considerations . . . . . . . . . . . . . . . . . . . . . . . . 523.5 Planning for connectivity support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3.5.1 Fibre Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.5.2 Fibre Channel over Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533.5.3 Quad Data Rate InfiniBand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Chapter 4. Installation and Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.1 First time installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.1.1 Installing the hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.2 Cabling the system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.2.1 Cabling for Fibre Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.2.2 FC port speed settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.2.3 Cabling for FCoE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.2.4 Cabling for QDR InfiniBand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.2.5 FC cable type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.2.6 Ethernet management cabling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.7 Power requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.8 Cooling requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.9 Cable Connector locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.3 Initializing the system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 654.3.1 About encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664.3.2 Prepare for initialization using InitTool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.3.3 Initializing the system through the web management interface . . . . . . . . . . . . . . 82

4.4 RAID Storage Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 974.4.1 Changing RAID modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

4.5 Connectivity Guidelines for Improved performance . . . . . . . . . . . . . . . . . . . . . . . . . . 1004.5.1 Interface card configuration guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1004.5.2 Host adapter guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1014.5.3 Cabling guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1014.5.4 Zoning guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Chapter 5. IBM FlashSystem 840 Client Host Attachment and Implementation . . . . 1035.1 Host implementation and procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.2 Host connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.3 Host attachment kit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.4 Operating system connectivity and best practices . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

5.4.1 FlashSystem 840 sector size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.4.2 File alignment for best RAID performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1055.4.3 IBM AIX and FlashSystem 840 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1055.4.4 FlashSystem 840 and Linux client hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1095.4.5 FlashSystem 840 and Windows client hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1115.4.6 FlashSystem 840 and client VMware ESX hosts . . . . . . . . . . . . . . . . . . . . . . . . 1155.4.7 FlashSystem 840 and Solaris client hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1165.4.8 FlashSystem 840 and HP-UX client hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118vi Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189TOC.fm5.4.9 FlashSystem 840 and IBM SAN Volume Controller / Storwize V7000 . . . . . . . . 1195.5 FlashSystem 840 Preferred Read and configuration examples . . . . . . . . . . . . . . . . . 119

5.5.1 FlashSystem 840 deployment scenario with preferred read. . . . . . . . . . . . . . . . 1205.5.2 Implementing preferred read. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1215.5.3 Example of Linux configuration file multipath.conf. . . . . . . . . . . . . . . . . . . . . . . . 1375.5.4 Example of VMWare configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

5.6 FlashSystem 840 and Easy Tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1425.7 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

5.7.1 Troubleshooting Linux InfiniBand configuration issues. . . . . . . . . . . . . . . . . . . . 1435.7.2 Linux fdisk error message. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1445.7.3 Changing FC port properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1455.7.4 Verifying alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Chapter 6. Using IBM FlashSystem 840 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1476.1 Overview of IBM FlashSystem 840 management tools. . . . . . . . . . . . . . . . . . . . . . . . 148

6.1.1 Access to the Graphical User Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1486.1.2 Graphical User Interface layout. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1496.1.3 Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1506.1.4 Multiple selections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1536.1.5 Status Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

6.2 Monitoring menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1546.2.1 Monitoring System menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1556.2.2 Monitoring Events menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1626.2.3 Monitoring Performance menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

6.3 Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1766.3.1 Navigating to Volumes Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1766.3.2 Volumes menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1776.3.3 Volumes by Host menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

6.4 Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1876.4.1 Navigating to Hosts Menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1876.4.2 Volumes by Host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

6.5 Access menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1946.5.1 Navigating to the Access menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1946.5.2 Users menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1956.5.3 Access CLI using PuTTY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1986.5.4 User Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2016.5.5 Audit Log menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Chapter 7. Configure Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2097.1 Settings menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

7.1.1 Navigating to the Settings Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2107.1.2 Event Notification menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2117.1.3 Directory Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2137.1.4 Network menu. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2247.1.5 Support menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2257.1.6 General menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

7.2 Service Assistant Tool. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2337.2.1 How to access the Service Assistant Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2337.2.2 Log in to the Service Assistant Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

Chapter 8. Product Integrations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2378.1 Running IBM FlashSystem 840 with SAN Volume Controller . . . . . . . . . . . . . . . . . . . 238

8.1.1 IBM System Storage SAN Volume Controller Introduction . . . . . . . . . . . . . . . . . 238 Contents vii

8.1.2 SAN Volume Controller hardware and software updates of interest to this book 243

8189TOC.fm Draft Document for Review February 7, 2014 7:33 pm8.1.3 SAN Volume Controller advanced storage functionality . . . . . . . . . . . . . . . . . . . 2468.1.4 Configuring IBM FlashSystem 840 for SAN Volume Controller . . . . . . . . . . . . . 249

8.2 IBM PureFlex System and IBM Storwize V7000 considerations. . . . . . . . . . . . . . . . . 2698.2.1 FlashSystem running in a PureFlex System environment . . . . . . . . . . . . . . . . . 2698.2.2 IBM Storwize V7000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

Chapter 9. Use Cases and Solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2759.1 Introduction to the usage cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2769.2 Tiering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277

9.2.1 Easy Tier or block level tiering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2779.2.2 ILM or File Level Tiering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

9.3 Preferred Read . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2799.3.1 Implementing preferred read. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

9.4 Flash Only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2869.5 Comparison. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Chapter 10. Hints and Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28910.1 Firmware upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29010.2 Service Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29110.3 Hints and tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

10.3.1 General guidelines for testing a specific configuration . . . . . . . . . . . . . . . . . . . 29210.3.2 Save Default Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29210.3.3 Test scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29310.3.4 Data Center Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29310.3.5 Secure Erase of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29310.3.6 Restoring FlashSystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29310.3.7 Performance data gathering basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

10.4 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29710.4.1 Troubleshooting pre-requisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29710.4.2 User interfaces for servicing your system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29910.4.3 Event reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30210.4.4 Resolving a problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305

10.5 IBM System storage Interoperation Center (SSIC) . . . . . . . . . . . . . . . . . . . . . . . . . . 305Appendix A. SAN Best Practices for 16 Gbps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30716 Gbps Fibre Channel Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308IBM System Storage b-type Gen 5 SAN - Product Overview. . . . . . . . . . . . . . . . . . . . . . . 309SAN Design Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Topologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315Inter-Switch Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316Inter-Chassis Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316Device placement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317Fan-In Ratios and Oversubscription . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318FCoE as ToR solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319Data Flow Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319Redundancy and Resiliency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321Distance and FCIP Design Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326Scalability and Supportability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328

Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329Initial Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335viii Implementing IBM FlashSystem 840 Storage

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335

Draft Document for Review February 7, 2014 7:33 pm 8189TOC.fmOther publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 Contents ix

8189TOC.fm Draft Document for Review February 7, 2014 7:33 pmx Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189pref.fmPreface

Almost all technological components in the data center are getting faster; central processing units, network, SAN, and memory. All of them have improved their speed by a minimum of 10X; some of them by 100X, for example the case of data networks. However, spinning disk has only increased its performance by 1.2 times.

Fortunately flash storage has come along and closed the gap between the other improvements in server technology and storage. Using IBM FlashSystem 840 provides value that goes beyond those benefits that are seen on disk-based arrays. These benefits include better user experience, server and application consolidation, development cycle reduction, application scalability, data center foot-print savings, and improved price performance economics.

This IBM Redbooks publication introduces clients to the newest member of IBM FlashSystem products; IBM FlashSystem 840. It will provide in-depth knowledge of the product architecture, software and hardware, its implementation, and hints and tips. Also illustrated are use cases that show real-world solutions for tiering, flash-only, and preferred read, as well as examples of the benefits gained by integrating FlashSystem storage into business environments.

Also discussed are product integration scenarios running FlashSystem 840 with IBM SAN Volume Controller, as well as IBM PureFlex System and IBM Storwize V7000 consideration when integrating with FlashSystem 840. We also describe best practice guidance for your FlashSystem environment with IBM 16 Gbps b-type products and features, focusing on Fibre Channel design.

This IBM Redbooks publication is intended for pre- and post-sales technical support professionals and storage administrators, and for anyone who wants to understand and learn how to implement this new and exciting technology. Copyright IBM Corp. 2013. All rights reserved. xi

8189pref.fm Draft Document for Review February 7, 2014 7:33 pmAuthorsThis book was produced by a team of specialists from around the world working at the International Technical Support Organization, Poughkeepsie Center.

Karen Orlando is a Project Leader at the International Technical Support Organization, Tucson Arizona Center. Karen has over 25 years in the IT industry with extensive experience in open systems management, and Information and Software development of IBM hardware and software storage. She holds a degree in Business Information Systems from the University of Phoenix and is Project Management Professional (PMP) certified since 2005.

Marcos Figueirdo Jr. is a technical Sales Leader in Latin American for FlashSystem, working with a distinct team in Growth Market Unit, that is focused ion FlashSystem solutions. Marcos previously worked as a Storage Specialist Consultant in Lab Services Brazil. With more than 5 years of experience in storage environments, he lead several assessments in Latin America. He holds a degree in Computer Science from the University of Brasilia. His areas of expertise include IBM Tivoli Storage Productivity Center, storage infrastructure consulting, design, implementation services, storage performance monitoring and tuning. He previous storage experience includes IBM DS8000, IBM Storwize V7000 and V7000 Unified, Scale-out NAS and Nseries.

Adrian Flores is a IBM FlashSystem Technical Sales Executive. Adrian has been involved in IT and Storage Technologies for over 18 years. He has performed innumerable roles including storage administration, technical support, solutions architect and technology principal. Adrian spent many years at technology leading companies such as ATT-Bell Labs in Mexico, Unix System Labs, Sun Microsystems, Perot Systems, Novus Consulting Group and EMC. Adrian's current role is to lead World Wide field technical-sellers for IBM Flash Storage products. Primary responsibilities are to provide technical sales enablement, competitive winning strategies and provide cohesive technical-sales solutions to a multitude of IBM internal organizations and IBM Customers World Wide.xii Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189pref.fmThanks to the following people for their contributions to this project:Bertrand DufrasneInternational Technical Support Organization, Poughkeepsie Center

Nathalie BickleyPhilip ClarkAdam ChunnJames CioffiJustin HaggardWoody HutsellMatt KeyKim Miller

Detlef Helmbrecht is an Advanced Technical Skills (ATS) IT Specialist working for the IBM Systems & Technology Group. He is located in the European Storage Competence Center (ESCC), Germany. Detlef has over 25 years of experience in IT performing numerous different roles including, software design, sales, and solution architect. His areas of expertise include high performance computing, disaster recovery, archiving, application tuning, and FlashSystem.

Ron Herrmann is an Executive Sales Specialist on the STG SSE Team. Ron has 30 years of experience in IT and came to IBM via Dilligent Technologies in 2008. Ron was a founding member of Diligent as well as several other start-ups which were acquired by EMC, Lucent, 3Com and Unisys. ERon was also an IT Director at a large Government Agency for five years. Ron is an expert in backup and disaster recovery, networking, and storage.

Carsten Larsen is an IBM Certified Senior IT Specialist working for the Technical Services Support organization in IBM Denmark delivering consultancy services to IBM customers within the storage arena. Carsten joined IBM in 2007, leaving behind a job at HP where he worked with Storage Arrays and UNIX for 10 years, holding, among others, the HP certification: Master Certified Systems Engineer (MASE).While working for IBM, Carsten has obtained Brocade BCFP and BCSD certifications as well as NetApp NCDA and NCIE certifications. Carsten is the author of a number of IBM Redbooks related to these product certifications.

Leandro Torolho is a SAN/Storage Specialist for IBM Global Services in Brazil. Since joining IBM in 2007, Leandro has been the SAN storage subject matter expert (SME) for many international customers. He is also an IBM Expert Certified IT Specialist. He holds a bachelor degree in computer science and he also has a post graduation degree in computer networks. Leandro's technical background also consists of UNIX (AIX) administration, and backup administration (Tivoli Storage Manager). Preface xiii

Michael Schwartz

8189pref.fm Draft Document for Review February 7, 2014 7:33 pmBobby SumnersRobert TillersonSteve VasquezIBM Houston Center, FlashSystem

Indulis BernsteinsIBM Sales & Distribution, STG Sales, IBM United Kingdom Limited

Martin TiernanIIBM Systems & Technology Group, Client Enablement & Systems Assurance, IBM Tucson

Now you can become a published author, too!

Heres an opportunity to spotlight your skills, grow your career, and become a published authorall at the same time! Join an ITSO residency project and help write a book in your area of expertise, while honing your experience using leading-edge technologies. Your efforts will help to increase product acceptance and customer satisfaction, as you expand your network of technical contacts and relationships. Residencies run from two to six weeks in length, and you can participate either in person or as a remote resident working from your home base.

Find out more about the residency program, browse the residency index, and apply online at:ibm.com/redbooks/residencies.html

Comments welcomeYour comments are important to us!

We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at:

ibm.com/redbooks

Send your comments in an email to:redbooks@us.ibm.com

Mail your comments to:IBM Corporation, International Technical Support OrganizationDept. HYTD Mail Station P0992455 South RoadPoughkeepsie, NY 12601-5400

Stay connected to IBM Redbooks Find us on Facebook:

http://www.facebook.com/IBMRedbooks

Follow us on Twitter:http://twitter.com/ibmredbooks

Look for us on LinkedIn:xiv Implementing IBM FlashSystem 840 Storage

http://www.linkedin.com/groups?home=&gid=2130806

Draft Document for Review February 7, 2014 7:33 pm 8189pref.fm Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks weekly newsletter:https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm

Stay current on recent Redbooks publications with RSS Feeds:http://www.redbooks.ibm.com/rss.html Preface xv

8189pref.fm Draft Document for Review February 7, 2014 7:33 pmxvi Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189spec.fmNotices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. Copyright IBM Corp. 2013. All rights reserved. xvii

8189spec.fm Draft Document for Review February 7, 2014 7:33 pmTrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol ( or ), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

AIXBigInsightsDB2developerWorksDiligentDS8000Easy TierFICONFlashCopyFlashSystem

GPFSIBMIBM FlashSystemIBM Flex SystemMicroLatencyNetViewPower SystemsPureFlexPureSystemsReal-time Compression

RedbooksRedbooks (logo) StorwizeSystem StorageSystem xTivoliTivoli Enterprise ConsoleVariable Stripe RAIDXIV

The following terms are trademarks of other companies:

Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Other company, product, or service names may be trademarks or service marks of others. xviii Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189intro.fmChapter 1. FlashSystem Storage Introduction

Flash technology in the data center is too relevant to be ignored for a few simple reasons: Since its introduction, flash storage has improved across all metrics offering: higher

performance, density, and reliability all of which translate to improved business efficiency. Flash cost per capacity and cost per transaction relative to hard disk based storage make

it very attractive to business' who are attempting to maintain pace in a 24x7 competitive marketplace.

Flash is easily integrated into existing data center environments and provides an instant boost to the mission critical applications.

While flash in storage is pervasive in the data center, its implementation varies considerably amongst competitors and technologies. Some use it as a simple cache accelerator while others implement it as yet another permanent data tier. The reality is that Flash only matters when two conditions in the data center are met: Flash eliminates IO bottlenecks while generating higher levels of application efficiency

(Improved performance) Storage Economics are improved by its use, this is, it provides lower TCO (Cost

Reduction) and faster ROI to the existing environment (enable new business opportunities).

IBM FlashSystem storage delivers high performance, efficiency, and reliability for shared enterprise storage environments. It helps clients address performance issues with their most important applications and infrastructure.

This chapter provides an introduction to IBM FlashSystem storage system and its core value, benefits and technological advantages.

1 Copyright IBM Corp. 2013. All rights reserved. 1

8189intro.fm Draft Document for Review February 7, 2014 7:33 pm1.1 FlashSystem Storage OverviewFlash technology has fundamentally changed the paradigm for IT systems, enabling new use cases and unlocking the scale of enterprise applications. Flash technology enhances the performance, efficiency, reliability and design of essential enterprise applications and solutions by addressing the bottleneck in the IT process (data storage), enabling truly optimized information infrastructure.

The IBM FlashSystem shared flash storage systems offer affordable, high-density, ultra low-latency, high-reliability and scalable performance in a storage device that is both space and power efficient. IBM Flash products, which can either augment or replace traditional hard-drive storage systems in enterprise environments, empower applications to work faster and scale further.

In addition to optimizing performance, the BM FlashSystem family brings enterprise reliability and macro efficiency to the most demanding data centers, allowing businesses to:

Reduce customer complaints by improving application response time Service more users with less hardware Reduce I/O wait and response times of critical applications Simplify solutions Reduce power and floor space requirements Speed up applications, thereby enhancing pace of business Improve utilization of existing infrastructure Compliment existing infrastructure Mitigate risk

From the client business perspective, an IBM FlashSystem provides focus benefits and value in four essential areas:

Extreme Performance Enable business to unleash the power of performance, scale, and insight to drive services and products to market faster.

MicroLatency Achieve competitive advantage through applications that enable faster decision making due to microsecond response times.

Macro Efficiency Decrease costs by getting more from efficient use of the IT staff, IT applications and IT equipment due to the efficiencies flash brings to the data center.

Enterprise Reliability Durable and reliable designs that use enterprise class flash and patented data protection technology

1.2 Why Flash mattersA recent International Data Corporation (IDC) study expects the flash market to grow at about 2 Implementing IBM FlashSystem 840 Storage

27% compounded annual rate from 2011 to 2016 and that the amount of Flash memory being

Draft Document for Review February 7, 2014 7:33 pm 8189intro.fmshipped is expected to grow by a factor of 20X reaching almost 3,000 PB or 3 Exabytes by 2016.

Clearly, this is a very vibrant, and fast growing technology. Clients are looking to solve data center problems, optimize applications, reduce costs and grow their businesses.

Here are some reasons why Flash is a must in every data center, and why an IBM FlashSystem changes the storage economics: Reduce application and server licensing costs, especially those related to databases and

virtualization solutions. Improve application efficiency, that is, an application's ability to process, analyze and

manipulate more information, faster. Improve server efficiency. Get more out of your existing processors, use less RAM per

server, and consolidate operations by having server resources spend more time processing data as opposed to waiting for data.

Improve storage operations. Eliminate costly application tuning, wasted developer cycles, storage array hot spots, array tuning, and complex troubleshooting. Decrease floor space usage and energy consumption by improving overall storage environment performance.

Best performance for critical applications by providing the lowest latency in the market.

Almost all technological components in the data center are getting faster: central processing units, network, SAN, and memory. All of them have improved their speeds by a minimum of 10X; some of them by 100X like the case of data networks. However, spinning disk has only increased its performance, by 1.2 times!

IBM FlashSystem 840 provides benefits that include better user experience, server and application consolidation, development cycle reduction, application scalability, data center foot-print savings, and improved price performance economics.

Flash improves performance of applications that are critical to the user experience such as market analytics and research applications, trading and data analysis interfaces, simulation, modeling, rendering and so forth. Server and application consolidation is possible due to the increased process utilization resulting from the low latency of flash memory which enables a server to load more users, more databases and more applications. Flash provides or gives back time for further processing within existing resources of such servers. Clients soon realize that there's no need to acquire or expand server resources as often or as soon as it was previously expected.

Development cycle reduction is possible as developers spend less time designing an application to work around the inefficiencies of hard disk drives and less time tuning for performance.

Data center footprint savings are due to high density and high performance per density flash solutions replacing racks of spinning hard disk drives. Reducing the data center footprint also translates into power and cooling savings making flash one of the greenest technologies for the data center

Note: Improved price : performance economics is due to the very low cost for performance from FlashSystem. The cost savings come from deploying fewer storage enclosures, fewer disk drives, fewer servers with fewer processors and less RAM while using less power, space and cooling. Flash is one of the best tools the data center manager has for improving data center economics.Chapter 1. FlashSystem Storage Introduction 3

8189intro.fm Draft Document for Review February 7, 2014 7:33 pm1.3 IBM FlashSystem family: Product Differentiation Flash is used widely in the data center, either within a server (PCIe cards or internal SSDs), in storage arrays (hybrid or all-flash), in appliances or platform bundles/solutions (hardware/software/network). Flash can be used as cache or as a data tier. The bottom line is that due to the vast and wide adoption of flash, there are a number of different flash architectures and therefore criteria that can be applied to compare flash options. See Figure 1-1.

Figure 1-1 The different deployments of Flash

Most storage vendors use and promote flash. The difference is how it is implemented, and the impact that such implementation has on the economics (cost reduction and revenue generation) for clients.Flash technology' is used to eliminate the storage performance bottleneck. The IBM FlashSystem family is the shared-storage market leader and it provides the lowest latency and consistent response times. It is purpose-built and designed from the ground up for flash.

Many of IBM's competitors in the market create flash appliances based on commodity server platforms and use software-heavy stacks. Many of these suppliers use hardware technologies designed and created for disk, not flash. Some hybrid arrays combine legacy storage designs, spinning disk (HDD) and solid state disk (SSD). These solutions that rely on SSD are by no means inferior as IBM storage portfolio include SSD and flash on a variety of storage platforms as well. However, these alternate solutions don't have the same low latency (MicroLatency) as the hardware accelerated FlashSystem.

IBM FlashSystem family versus SSD-based storage arraysFlash memory technologies appeared in the traditional storage systems some time ago. These SSD-based storage arrays help to successfully address the challenge of increasing I/Os per second needed by applications, and the demand for lower response times in particular tasks. An implementation example is the IBM Easy Tier technology. Refer to, Easy Tier on page 247 for an overview of this technology.

However, these technologies typically rely on flash in the format of Fibre Channel (FC), Serial attached storage (SAS), or serial advanced technology attachment (SATA) disks, placed in the same storage system as traditional spinning disks, and utilizing the same resources and

HW-based Vs SW-based architecturePurposed-built for flash or not?Flash Chip ChoicesData Protection SchemesScalability, Reliability, AvailabilityMethod of DeploymentFunctionality and FeaturesHybrid or All FlashAnd yes Latency!

HW-only data pathFPGA

IBM FlashSystemj4 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189intro.fmdata paths. This approach can limit the advantages of flash technology due to the limitations of traditional disk storage systems.

For more information on IBM Easy Tier, refer to the following IBM Redbooks publications: Implementing the IBM System Storage SAN Volume Controller V6.3, SG24-7933, Chapter

7, Easy Tier. IBM System Storage SAN Volume Controller Best Practices and Performance Guidelines,

SG24-7521, Chapter 11, IBM System Storage Easy Tier function IBM System Storage DS8000 Easy Tier, REDP-4667

IBM FlashSystem storage provides a hardware-only data path that realizes all of the potential of flash memory. These systems are different from traditional storage systems, both in the technology and usage.

An SSD device with a HDD disk form factor has flash memory that is put into a carrier or tray. This carrier is inserted into an array like a hard disk drive. The speed of storage access is limited by the following technology because it adds latency and cannot keep pace with flash technology: Array controllers and software layers SAS controllers and shared bus Tiering and shared data path Form factor enclosure

IBM FlashSystem products are fast and efficient. The hardware-only data path has a minimum number of software layers, which are mostly firmware components, as well as management software which is separated from the data path (out-of-band). The only other family of products that have hardware-only access to flash technology are PCI express (PCIe) flash products which are installed into a dedicated server. With the appearance of IBM FlashSystem, the benefits of PCIe flash products to a single server can now be shared by many servers.

1.4 Technology and Architectural Design OverviewIBM FlashSystem, with an all-hardware data path using field programmable-gate array (FPGA) modules, are engineered to deliver the lowest possible latency. They incorporate proprietary flash controllers and leverage numerous patented technologies. FlashSystem controllers have proprietary logic design, firmware, and system software. There are no commodity 2.5-inch SSDs, PCIe cards, or any other significant non-IBM assemblies within the system. The flash chips, FPGA chips, processors, and other semiconductors in the system are carefully selected to be consistent with the purpose-built design, which is designed from the ground up for high performance, reliability, and efficiency. Notable architectural concepts of the IBM FlashSystem storage systems are: Hardware-only data path Leverage FPGA's extensively Field-upgradable hardware logic Less expensive design cycle Extremely high degree of parallelism Intelligent flash modulesChapter 1. FlashSystem Storage Introduction 5

8189intro.fm Draft Document for Review February 7, 2014 7:33 pm Distributed computing model Low-power PPC processors Interface and flash processors run thin real time operating systems. Management processor communicates with the interface and flash processors via internal

network Minimal management communication

Hardware-only data pathThe hardware-only data path design of IBM FlashSystem eliminates the software layer latency that is found in other vendor products. In order to achieve such extremely low latencies, IBM FlashSystem advanced software functions are carefully assessed and implemented on a limited basis. For environments requiring advanced storage services, implementing IBM FlashSystem with IBM SAN Volume Controller can offer an unmatched combination of performance, low latency, and rich software functionality.

In IBM FlashSystem and data traverses the array controllers through FPGAs and dedicated, low-power CPUs. There's no wasted cycles on interface translation, protocol control or tiering.

IBM FlashSystem, with an all-hardware data path design, have an internal architecture that is different from the other hybrid (SSD + HDD) or SSD-only based disk systems.

Flash chipsThe flash chip is the basic storage component of the flash module. There can be a maximum of 80 enterprise multi-level cell (eMLC) flash chips per flash module. Combining flash chips of different flash technologies is not supported in the same flash module or storage system to maintain consistent wearing and reliability.

Gateway interface FPGAThe gateway interface FPGA is responsible for providing I/O to the flash module and direct memory access (DMA) path. It is located on the flash module and has two connections to the backplane.

Flash controller FPGAThe flash controller FGPA of the flash module is used to provide access to the flash chips and is responsible for the following functions: Provide data path, hardware I/O logic Look up tables and write buffer Control 20 flash chips Operate independently of other controllers Maintain write ordering and layout Provide write setup Maintain garbage collection Provide error handling

Figure 1-2 shows the flash Controller Design details.6 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189intro.fmFigure 1-2 IBM FlashSystem controller details

The concurrent operations performed on the flash chips include moving data in and out of the chip via DMA, and by internally moving data and performing erases. This means that while actively transferring user data in the service of host initiated I/O, the system can be simultaneously running garbage collection activities without impacting the I/O. The ratio of transparent background commands running concurrent to active data transfer commands is seven to one.

There are a maximum of four flash controllers per flash module: two per primary board and two per expansion board .

1.4.1 Variable Stripe RAID and two dimensional Flash RAID OverviewStorage systems of any kind are typically designed to perform two main functions: to store and protect data. IBM FlashSystem include the following options for data protection. Table 1-1 shows the various methods of protection. RAID data protection

Variable Stripe RAID Two dimensional (2D) Flash RAID

Flash memory protection methods Optimized RAID Rebuild times

Flash Controller - FPGAData path, Hardware I/O logic Look up Tables and Write BufferControls 20 flash Chips

Lookup TablesDRAM

Control PPC and DRAM Out of Data path operationsGarbage collection, Error Handling, System HealthWear Leveling, Statistics, etc.

DRAM Write Buffer

Gateway Interface FPGAI/O and Direct Memory Access

NAND Flash Memory

FPGAFPGAChapter 1. FlashSystem Storage Introduction 7

8189intro.fm Draft Document for Review February 7, 2014 7:33 pmTable 1-1 Various IBM FlashSystem protection

1.5 Variable Stripe RAIDVariable Stripe RAID or VSR is a unique IBM technology that provides data protection of the memory page, block, or whole chip, which eliminates the necessity to replace a whole flash module in a case of a single memory chip failure or plane failures. This in turn, expands the life and endurance of Flash Modules and reduces considerably maintenance events throughout the life of the system.

VSR provides high redundancy across chips within a flash module. RAID is implemented at multiple addressable segments within chips, in a 9+1 RAID 5 fashion, and it is controlled at the flash Controller level (four in each flash module). Due to the massive parallelism of DMA operations controlled by each FPGA and parallel access to chip sets, dies, planes, blocks and pages; the implementation of VSR has minimum impact on performance.

The following information describes some of the most important aspects of VSR implementation: VSR is managed and controlled by each of the four flash Controllers within a single

module. A flash Controller is in charge of only 20 flash chips Data is written on flash pages of 8 KB and erased in 1MB flash blocks VSR is implemented/managed at flash chip plane levels There are 16 planes per chip Before a plane fails, at least 256 flash blocks within a plane must be deemed failed A plane can also fail in its entirety Up to 64 planes can fail before a whole module is considered failed Up to 4 chips can fail before a whole module is considered failed When a flash Module is considered failed, 2D Flash RAID takes control of data protection

and recovery

LAYER MANAGED BY PROTECTION

System-level RAID 5 Centralized RAID Controllers

Module failure

Module-level RAID 5 Each module across the chips

Chip failure, page failure

Module-level Variable Stripe RAID

Each module across the chips

Subchip, chip, or multi-chip failure

Chip-level ECC Each module using the chips

Bit and block error

Note: The proprietary two dimensional (2D) Flash RAID data protection scheme of the IBM FlashSystem 840 storage system combines system-level RAID 5 and Module-level Variable Stripe RAID (not just module level RAID).8 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189intro.fm When a plane or a chip fails, VSR activates to protect data while maintaining system-level performance and capacity.

1.6 How VSR worksVariable Stripe RAID is an IBM patented technology. It includes, but is more advanced than a simple RAID of flash chips. Variable Stripe RAID introduces two key concepts: 1. The RAID stripe is not solely across chips, it actually spans across flash layers.2. The RAID stripe can automatically vary based on observed flash plane failures within a

flash module. For example, stripes are not fixed at 9+1 RAID 5 stripe members, but can go down to 8+1, 7+1 or even 6+1 based on plane failures.

This ability to protect the data at variable stripes effectively maximizes flash capacity even after flash component failures. Figure 1-3 and shows an overview of IBM FlashSystem Variable Stripe RAID (VSR)

Figure 1-3 IBM FlashSystem Variable Stripe RAID (VSR)

Figure 1-4 on page 10 shows the benefits of IBM VSR.

Patented VSR allows RAID stripe sizes to vary. If one die fails in a ten-chip stripe, only the failed die is bypassed, then data is

restriped across the remaining nine chips. VSR reduces maintenance intervals caused by flash failures

16 Planes

10 Chips

FAILFAILFAILChapter 1. FlashSystem Storage Introduction 9

8189intro.fm Draft Document for Review February 7, 2014 7:33 pmFigure 1-4 The Value of IBM FlashSystem Variable Stripe RAID.

It is important to emphasize that VSR only has an effect at the plane level. This means that only the affected planes within a plane failure are converted to (N-1), yet it will maintain the current stripe member count (9+1) layout through the rest of the areas of all other planes and chips that are not involved in the plane failure.

To illustrate how VSR functions, assume that a plane fails within a flash chip and is no longer available to store data. This might occur as a result of a physical failure within the chip, or some damage being inflicted on the address or power lines to the chip. The plane failure would be detected and the system could change the format of the page stripes that are used. The data that was previously stored in physical locations across chips in all ten lanes using a page stripe format with ten pages, is now stored across chips in only nine lanes using a page stripe format with nine pages as reflected. Thus, no data stored in the memory system was lost, and the memory system can self-adapt to the failure and continue to perform and operate by processing read and write requests from host devices.

This ability of the system to automatically self-adapt, when needed, to chip and intra-chip failures makes the FlashSystem flash module extremely rugged and robust, and capable of operating despite the failure of one or more chips or intra-chip regions. It also makes the system very user-friendly in that the failure of one, two, or even more individual memory chips or devices does not require the removal and potential disposal of previously used memory storage components. The reconfiguration or reformatting of the data to change the page stripe formatting to account for chip or intra-chip failures might reduce the amount of physical memory space that is held in reserve by the system and available for the system for background operation. It should be noted that in all but the most extreme of circumstances (in which case the system will create alerts), it does not impact usable capacity of performance.

Reliability, availability, and serviceabilityThe previous explanation brings other important topics to discuss, one of them being an increase in RAS levels or Reliability, Availability and Serviceability levels. And more importantly, IBM FlashSystem RAS levels over other technologies.

Following is a summary of the capabilities of VSR:

Form Factor SSD Flash failure = Disk failure Requires top-level RAID Relatively frequent hot-swaps

Enterprise Flash Drive or Memory Module Flash failure = Degraded state

within module Performance impact on RAID set Hot-swap to resolve

FlashSystem with Variable Stripe RAID Preserves flash life Preserves performance Re-parity data in microseconds

No Parity

Value of Variable Stripe RAID

Less maintenance touches while still preserving the life, protection, and performance of the Day-1 experience

Parity

Parity10 Implementing IBM FlashSystem 840 Storage

Patented Variable Stripe RAID allows RAID stripe sizes to vary.

Draft Document for Review February 7, 2014 7:33 pm 8189intro.fm If one plane fails in a ten-chip stripe, only the failed Plane is bypassed, and then data is restriped across the remaining nine chips. No system rebuild is needed

VSR reduces maintenance intervals caused by flash failures.

1.7 Two dimensional (2D) Flash RAIDTwo dimensional (2D) Flash RAID refers to the combination of Variable Stripe RAID (at the flash module level) and system-level RAID 5.The second Dimension of data protection, hence the name two-dimensional Flash RAID is actually implemented going across Flash modules of RAID 5 protection. This system level RAID 5 is striped across the appropriate number of flash modules in the system based on the selected configuration. System-level RAID-5 could stripe across four (2D+1P+1S), eight (6D+1P+1S) or twelve flash modules (10D+1P+1S). The architecture allows to designate a dynamic flash module Hot Spare. Figure 1-5 shows IBM FlashSystem 2D RAID as illustrated in Figure 1-5.

Figure 1-5 IBM FlashSystem 2D-RAID

Two dimensional (2D) Flash RAID technology within IBM FlashSystem provides two independent layers of RAID 5 data protection within each system as mentioned earlier. The module-level Variable Stripe RAID technology, and an additional system-level RAID 5 across flash modules. When operating in system-level RAID 5 mode, redundant centralized RAID controllers create a stripe arrangement across the four, eight or twelve flash modules in the system. The system-level RAID 5 complements the Variable Stripe RAID technology implemented within each flash module, and provides protection against data loss and data unavailability resulting from flash module failures. It also allows data to be rebuilt onto a hot spare Flash module, so flash modules can be replaced without data disruption.

In addition to 2D Flash RAID and Variable Stripe RAID data protection, IBM FlashSystem family storage systems incorporate other reliability features, including: Error correcting codes to provide bit-level reconstruction of data from flash chips.

RAID 5 across Flash Modules (10 data + 1 parity + 1 hot spare)

ExternalInterfaces

(FC, IB)RAID

Controllers

VSR within Flash Modules

(9 data + 1 parity)2D Flash RAID

Interface AInterface AInterface A Interface BInterface BInterface B

RAID Controller ARAID Controller oonC ARAID Controller A RAID Controller BRAID Controller oon BRAID Controller BChapter 1. FlashSystem Storage Introduction 11

8189intro.fm Draft Document for Review February 7, 2014 7:33 pm Checksums and data integrity fields designed to protect all internal data transfers within the system.

Overprovisioning to enhance write endurance and decrease write amplification. Wear leveling algorithms balance the number of writes among flash chips throughout the

system. Sweeper algorithms to help ensure all data within the system is read periodically to avoid

data fade issues.

Understanding 2D Flash RAID allows the reader to visualize the tremendous advantage over other flash storage solutions. Both VSR and 2D Flash RAID are implemented and controlled at FPGA hardware based levels, and not via Software-heavy, external controllers and legacy architectures. Two-dimensional flash RAID eliminates single points of failure and provides enhanced system-level reliability. 12 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189FlashSystem-840.fmChapter 2. IBM FlashSystem 840 Architecture

This chapter describes the IBM FlashSystem 840 architecture in detail. Sections include an introduction to IBM FlashSystem 840, product features, a comparison to IBM FlashSystem 820, and an overview of the architecture and hardware. This chapter also provides an overview of administration and serviceability of IBM FlashSystem 840, interoperability, and integration with other IBM products.

For more details on IBM FlashSystem architecture, refer to the IBM FlashSystem 840 Infocenter website at:http://pic.dhe.ibm.com/infocenter/flshsstm/cust/index.jsp

2 Copyright IBM Corp. 2013. All rights reserved. 13

8189FlashSystem-840.fm Draft Document for Review February 7, 2014 7:33 pm2.1 Introduction to IBM FlashSystem 840IBM FlashSystem 840 is an all-flash storage array that provides extreme performance and large capacity while also providing enterprise class reliability and green data center power and cooling requirements. IBM FlashSystem 840 holds up to twelve 4 TB flash modules in only 2U of rack space, making it the densest all-flash storage array on the market. IBM FlashSystem 840 also provides up to 1,100,000 IOPS performance, up to 8 GBps bandwidth and latency as low as 90 microseconds, making it the perfect product for demanding customer workloads. This high capacity and extreme performance is also protected by IBM FlashSystem patented reliability technologies. IBM FlashSystem 840 supports several protocols including Fibre-channel (16, 8, and 4 Gbps), Fibre-channel over Ethernet (FCoE) and InfiniBand; enabling connections to high performance servers and storage area networks. The following sections describe IBM FlashSystem 840 core attributes. Figure 2-1 on page 14 shows the front view of the IBM FlashSystem 840.

Figure 2-1 IBM FlashSystem 840 front view

2.1.1 IBM FlashSystem 840 capacityIBM FlashSystem 840 supports a maximum of twelve 4 TB flash modules which provides a maximum capacity of 48 TB (RAID 0). IBM FlashSystem 840 can be ordered with two, four, eight, or twelve flash modules. The flash modules available are either 2 TB or 4 TB storage capacity.

IBM FlashSystem 840 supports both RAID 0 and RAID 5 configurations.

IBM FlashSystem 840 supports the creation of up to 2048 LUNs. The size of the LUNs can be from 1 MiB to 45 TiB in size (not to exceed total system capacity). IBM FlashSystem 840 supports the connection of up to 1024 hosts per interface port and allows the mapping of multiple LUNs to each host. Table 2-1 on page 15 lists all the combinations of storage capacities for the various configurations of IBM FlashSystem 840.

Note: Two TB and 4 TB Flash modules cannot be intermixed in the same IBM FlashSystem 840 chassis.

Note: The maximum usable capacity of IBM FlashSystem 840 in RAID 0 mode is 45 TiB. The maximum usable capacity of IBM FlashSystem 840 in RAID 5 mode is 37.5 TiB. 14 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189FlashSystem-840.fmTable 2-1 IBM FlashSystem 840 Capacity in TB and TiB for RAID 0 and RAID 5.

2.1.2 IBM FlashSystem 840 performance and latencyIBM FlashSystem 840 utilizes all hardware Field-programmable gateway array (FPGA) components in the data path which enables the fastest possible IO rates and lowest latency. IBM FlashSystem 840 provides extreme performance of up to 1,100,000 IOPS and up to 8 GBps in bandwidth. IBM FlashSystem 840 provides write latency as low 90s and read latency as low as 135s.Table 2-2 on page 16 illustrates IBM FlashSystem performance at various IO patterns.

IBM FlashSystem 840 Configuration

RAID 0 TB RAID 5 TB RAID 0 TiB RAID 5 TiB

Two 2 TB Flash Modules 4 N/A 3.75 N/A

Four 2 TB Flash Modules 8 4 7.5 3.75

Eight 2 TB Flash Modules 16 12 15 7.5

Twelve 2 TB Flash Modules 24 20 22.5 18.75

Two 4 TB Flash Modules 8 N/A 7.5 N/A

Four 4 TB Flash Modules 16 8 15 7.5

Eight 4 TB Flash Modules 32 24 30 22.5

Twelve 4 TB Flash Modules 48 40 45 37.5

Note: The following are the exact byte counts for the flash modules used in IBM FlashSystem 840:

4 TB module:Presented: 4123162312704 bytes (after VSR and reserve)Raw Flash: 5497558138880 bytes (80 chips * 64 GiB)IBM FlashSystem 840 GUI reports 3.75 TiB usableTherefore, the actual maximum raw flash capacity of an IBM FlashSystem 840 system is 5497558138880 bytes x 12 or approximately 66 TB.

2 TB module:Presented: 2061581156352 bytes (after VSR and reserve)Raw Flash: 2748779069440 bytes (40 chips * 64 GiB)IBM FlashSystem 840 GUI reports 1.875 TiB usableChapter 2. IBM FlashSystem 840 Architecture 15

8189FlashSystem-840.fm Draft Document for Review February 7, 2014 7:33 pmTable 2-2 IBM FlashSystem 840 performance at various IO pattens.

2.1.3 IBM FlashSystem 840 power requirementsIBM FlashSystem 840 is green data center friendly. IBM FlashSystem 840 only consumes 625 W of power (steady state RAID 5 configuration for a 30/70 write/read workload on an 8 module 2 TB flashcard system) and uses two standard single phase (100v - 240v) electrical outlets.

2.1.4 IBM FlashSystem 840 physical specificationsIBM FlashSystem 840 installs in a standard 19 equipment rack. IBM FlashSystem 840 is 2U high and 19 wide. A standard data 42U 19 data center rack can be fully populated with 21 IBM FlashSystem 840 systems.

The physical dimensions of IBM FlashSystem 840 are: Height: 3.5 (8.90 mm) Width: 19 (48 mm) Length: 31.4 (79 mm) Weight (maximum configuration - 12 Flash Modules): 75 lbs (34.02 kg) Airflow path: cool air flows into front of unit (intake) to rear of unit (exhaust)

Performance Criteriaab

a. Data gathered using an Oakgate storage test appliance and a Fibre Channel protocol analyzerb. All measurements are made in a RAID 5 configuration, 4 TB cards, 90% of usable capacity

Maximum Capacity (12 Flash modules)

Mid Capacity(8 flash modules)

Minimum Capacity (4 flash modules)

100% Read IOPS 1.1 M 1.1M 1.0 M

100% Write IOPS 600 K 400 K 225 K

70/30 IOPS 750 K 500 K 225 K

100% Large Block Sequential Read

8 GB/sec 8 GB/sec 4 GB/sec

100% Large Block Sequential Write

4 GB/sec 2.5 GB/sec 1 GB/sec

Read Latency 135 s 135 s 135 sWrite Latency 90 s 90 s 90 s

Notes: Plan to attach each of the two power supplies in the enclosure to separate main power supply lines.Plan to connect the power cords on the right side of the rack (when viewed from the rear) to power sources that provides power in the 100V - 240V ac range. Each power supply requires 900 watts (start-up rating). Using two power sources provides power redundancy.

Attention: The power cord is the main power disconnect. Ensure that the socket outlets are located near the equipment and are easily accessible.16 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189FlashSystem-840.fm Heat: 2180 BTU (maximum configuration RAID 5)

2.1.5 IBM FlashSystem 840 reliability and serviceabilitySimilar to all IBM FlashSystem products, IBM FlashSystem 840 provides enterprise class reliability and serviceability that is unique for all-flash storage arrays. IBM FlashSystem 840 uses the following technologies for data protection and maximum system up-time: Block remapping: Assures that flash cells are protected from adjacent activity Flash Cell leveling: A technology that reduces flash cell wear due to electrical

programming. Two dimensional RAID (2D) Flash RAID: Allows the provisioning of an entire flash module

as a spare to be used in the event of another flash module failure. Variable Stripe RAID (VSR): A patented IBM technology that provides an intra module

RAID stripe on each flash module.

New reliability and serviceability features of IBM FlashSystem 840In addition to the standard features above, IBM FlashSystem 840 includes the following new reliability and seviceability features: Hot swappable flash modules via the front panel. In the event of a flash module failure,

critical customer applications can remain online while the defective module is replaced.Since customer application downtime does not need to be scheduled, this service can typically be performed immediately versus having to wait days for a service window. A simple CLI command is all that is needed to prepare IBM FlashSystem 840 for a flash module replacement. The flash modules can be easily removed from the front of IBM FlashSystem 840 unit, without the need to remove top access panels or extend cabling.

Concurrent code loads. IBM FlashSystem 840 supports concurrent code load, enabling customer applications to remain online during firmware upgrades to all components, including the flash modules.

Redundant hot swappable components. RAID controllers called canisters, management modules, interface cards (all contained in the canister), batteries, fans, and power supplies are all redundant and hot swapable. All components are easily accessible via the front or rear of the unit so IBM FlashSystem 840 does not need to be moved in the rack, nor any top access panels or cables need to be extended. This makes servicing the unit extremely easy.

2.1.6 IBM FlashSystem 840 scalabilityIBM FlashSystem 840 supports the ability to grow the storage capacity after deployment. IBM FlashSystem 840 supports a maximum configuration of twelve 2 TB or 4 TB flash modules. IBM FlashSystem 840 can be purchased with two, four, eight, or twelve 2 TB or 4 TB modules.

Tip: Concurrent code loads require that all connected hosts have at least two connections, at least one to each canister, to FlashSystem 840. See section 10.1, Firmware upgrade on page 290, for more information.

Tip: It is possible to buy an entry level IBM FlashSystem 840 unit with only two flash modules. This unit can only be configured as RAID 0. It can be expanded to 4, 8, or 12 modules and can then be reconfigured as RAID 5.Chapter 2. IBM FlashSystem 840 Architecture 17

8189FlashSystem-840.fm Draft Document for Review February 7, 2014 7:33 pmIBM FlashSystem 840 upgrade options include: Systems that are purchased with four 2 TB flash modules, can be expanded to eight or

twelve 2 TB flash modules. Systems that are purchased with four 4 TB flash modules, can be expanded to eight or

twelve 4 TB flash modules. Systems that are purchased with eight 2 TB flash modules, can be expanded to twelve 2

TB flash modules. Systems that are purchased with eight 4 TB flash modules, can be expanded to twelve 4

TB flash modules

2.1.7 IBM FlashSystem 840 protocol supportIBM FlashSystem 840 supports the following interface protocols and number of connections: Fibre-Channel (16 ports of 4 Gbps or 8 Gbps) Fibre-Channel (8 ports of 16 Gbps (these ports also support 8 Gbps and 4 Gbps) Fibre-Channel over Ethernet (FCoE) (16 ports of 10 Gbps FCoE) InfiniBand (8 ports of QDR 40 Gbps)

2.1.8 IBM FlashSystem 840 encryption support IBM FlashSystem 840 provides optional encryption of data at rest, which protects against the potential exposure of sensitive user data and user metadata that is stored on discarded or stolen flash modules. Encryption of system data and metadata is not required, so system data and metadata are not encrypted.

Notes: Is is not possible to mix 2 TB and 4 TB flash modules in any configuration on IBM

FlashSystem 840. If an IBM FlashSystem 840 is purchased with 2 TB flash modules, all system

expansions must be with 2 TB flash modules. If an IBM FlashSystem 840 is purchased with 4 TB flash modules, all system

expansions must be with 4 TB flash modules. Expanding an IBM FlashSystem 840 unit with two, four or eight additional flash modules

requires that the system be reconfigured. A backup of the system configuration and data migration, if needed, should be planned before the expansion.

Notes: IBM FlashSystem 840 only supports one interface type per system. For example, it is

not possible to have two Fibre-channel interface cards and two InfiniBand interface cards in the same IBM FlashSystem 840 system.

IBM FlashSystem 840 supports eight active ports across the entire system if 16 Gbps fibre-channel is enabled. These eight ports can operate at 16, 8 or 4 Gbps.18 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189FlashSystem-840.fmThe following topic describes general encryption concepts and terms for IBM FlashSystem 840: Encryption-capable refers to the ability of IBM FlashSystem 840 to optionally encrypt user

data and metadata by using a secret key. Encryption-disabled describes a system where no secret key is configured. The secret key

is neither required, nor used, to encrypt or decrypt user data. Encryption logic is actually still implemented by IBM FlashSystem 840 while in the encryption-disabled state, but uses a default, or well-known, key. Therefore, in terms of security, encryption-disabled is effectively the same as not encrypting at all.

Encryption-enabled describes a system where a secret key is configured and used. This does not necessarily mean that any access control has been configured to ensure that the system is operating securely. Encryption-enabled only means that the system is encrypting user data and metadata using the secret key.

Access-control-enabled describes an encryption-enabled system that is configured so that an access key must be provided to authenticate with an encrypted entity, such as a secret key or flash module, to unlock and operate that entity. IBM FlashSystem 840 permits access control enablement only when it is encryption-enabled. A system that is encryption-enabled can optionally also be access-control-enabled to provide functional security.

Protection-enabled describes a system that is both encryption-enabled and access-control-enabled. An access key must be provided to unlock IBM FlashSystem 840 so that it can transparently perform all required encryption-related functionality, such as encrypt on write and decrypt on read.

The Protection Enablement Process (PEP) transitions IBM FlashSystem 840 from a state that is not protection-enabled to a state that is protection-enabled. The PEP requires that the customer provide a secret key to access the system, and the secret key must be resiliently stored and backed up externally to the system; for example, on a USB memory stick. PEP is not merely activating a feature via the GUI or CLI. To avoid loss of data that was written to the system before the PEP occurs, the customer must move all of the data to be retained off of the system before the PEP is initiated, and then must move the data back onto the system after PEP has completed. The PEP is performed during the system initialization process, if encryption is activated.

Application-transparent encryption is an attribute of IBM FlashSystem 840 encryption architecture, referring to the fact that applications are not aware that encryption and protection is occurring. This is in contrast to Application Managed Encryption (AME), where an application must serve keys to a storage device.

Note: While some IBM products which implement encryption of data at rest stored on a fixed block storage device implement encryption using self-encrypting disk drives (SEDs), IBM FlashSystem 840 flash module chips do not use SEDs. IBM FlashSystem 840 data encryption and decryption is performed by the flash modules, which can be thought of as the functional equivalent of Self-Encrypting Flash Controller (SEFC) cards.

Note: IBM FlashSystem 840 requires a license for encryption. If encryption is required, validate with IBM Sales or your IBM Business Partner that the license is ordered with the equipment. Chapter 2. IBM FlashSystem 840 Architecture 19

8189FlashSystem-840.fm Draft Document for Review February 7, 2014 7:33 pmConfiguring encryptionYou can use the initialization tool encryption activation option to activate encryption when an encryption feature code is purchased. If encryption is activated, an encryption key is generated by the system to be used for access to the system. The tool invokes a wizard that guides the user through the process of copying the encryption key to multiple USB keys.

Accessing an encrypted systemAt system startup (power on) or to access an encrypted system, the encryption key must be provided by an outside source so that IBM FlashSystem 840 can be accessed. The encryption key is provided by inserting of the USB memory sticks that were created during system initialization into a canister.

Encryption technologyKey encryption is protected by an Advanced Encryption Standard (XTS-AES) algorithm key wrap using the 256-bit symmetric option in XTS mode, as defined in the IEEE1619-2007 standard. An HMAC-SHA256 algorithm is used to create a hash message authentication code for corruption detection, and is itself additionally protected by a system-generated cyclic redundancy check (CRC).

2.1.9 IBM FlashSystem Models 820 and 840 comparisonThe following Table 2-3 on page 20 lists the differences between IBM FlashSystem 820 and IBM FlashSystem 840 modules.

Table 2-3 IBM FlashSystem 820 and 840 comparison

Important: All data stored on the flash modules must be erased before encryption can be activated.

Feature to compare IBM FlashSystem 820 IBM FlashSystem 840

Storage Capacity options (TB)

10, 12, 20, 24 4,8,12,16,20,24,32,40,48

Form Factor 1U 2U

Performance (IOPS) 525,000 1,100,000Bandwidth 5 GB/s 8GB/s

Latency (read/write) 110s / 25s 135s / 90sAvailable Interfaces 8 and 4 Gb/s Fibre-channel

40 Gb/s QDR InfiniBand16, 8 and 4Gb/s Fibre-channel40 Gb/s QDR InfiniBand10 Gb/s FCoE

Chip Type eMLC eMLC

Chip RAID Yes Yes

System RAID Yes Yes

Power Consumption (steady state RAID 5)

300W 625Wa20 Implementing IBM FlashSystem 840 Storage

Draft Document for Review February 7, 2014 7:33 pm 8189FlashSystem-840.fm2.1.10 IBM FlashSystem 840 ManagementIBM FlashSystem 840 includes state-of-the-art IBM storage management interfaces. IBM FlashSystem 840 Graphical Users Interface (GUI) and Command Line Interface (CLI) have been updated from previous versions of IBM FlashSystem products to include the IBM SAN Volume Controller (SVC) CLI and the new IBM SVC Graphical User Interface, which resembles the popular XIV GUI.

IBM FlashSystem 840 also uses a USB key for system initialization, similar to the IBM V7000 disk system. IBM FlashSystem 840 also supports SNMP, e-mail notification (SMTP) and syslog redirection.

Figure 2-2 on page 21 shows the IBM FlashSystem 840 GUI. See 2.3.2, IBM FlashSystem 840 system management on page 33 for more details of the use of the FlashSystem 840 GUI and CLI.

Figure 2-2 IBM FlashSystem 840 Graphical User Interface

LUN Masking Yes Yes

Management CLIHTML 3.0 c2001 GUI InterfaceSNMP

IBM SVC CLIHTML 5.0 c2011 IBM Storage GUISNMPE-mail alertsSyslog redirect

a. 625W is for a 30/70 write/read workload on an 8 module 2 TB flashcard system

Feature to compare IBM FlashSystem 820 IBM FlashSystem 840Chapter 2. IBM FlashSystem 840 Architecture 21

8189FlashSystem-840.fm Draft Document for Review February 7, 2014 7:33 pm2.2 IBM FlashSystem 840 architectureThis section describes IBM FlashSystem architecture. Key product design characteristics including performance, reliability and serviceability are discussed. Hardware components are also described.

2.2.1 IBM FlashSystem 840 architecture overviewThe design goals for IBM FlashSystem 840 are to provide the customer with the fastest and most reliable all flash storage array on the market, while making it simple to service and support with as little downtime as possible. IBM FlashSystem 840 uses many FPGA components and as little software as possible, keeping IO latency to a minimum and IO performance to a maximum.

Figure 2-3 on page 23 illustrates IBM FlashSystem 840 design. At the core of the system are the two high speed non-blocking crossbar buses. The crossbar buses provide two high speed paths, which carry the data traffic, and can be used by any host entry path into the system. There is also a slower speed b