IBM Spectrum ScaleVersion 5.0.4

Administration Guide

IBM

SC28-3102-02

Note

Before using this information and the product it supports, read the information in “Notices” on page863.

This edition applies to version 5 release 0 modification 4 of the following products, and to all subsequent releases andmodifications until otherwise indicated in new editions:

• IBM Spectrum Scale Data Management Edition ordered through Passport Advantage® (product number 5737-F34)• IBM Spectrum Scale Data Access Edition ordered through Passport Advantage (product number 5737-I39)• IBM Spectrum Scale Erasure Code Edition ordered through Passport Advantage (product number 5737-J34)• IBM Spectrum Scale Data Management Edition ordered through AAS (product numbers 5641-DM1, DM3, DM5)• IBM Spectrum Scale Data Access Edition ordered through AAS (product numbers 5641-DA1, DA3, DA5)• IBM Spectrum Scale Data Management Edition for IBM® ESS (product number 5765-DME)• IBM Spectrum Scale Data Access Edition for IBM ESS (product number 5765-DAE)

Significant changes or additions to the text and illustrations are indicated by a vertical line (|) to the left of the change.

IBM welcomes your comments; see the topic “How to send your comments” on page xxxv. When you send information toIBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate withoutincurring any obligation to you.© Copyright International Business Machines Corporation 2015, 2020.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract withIBM Corp.

Contents

Tables................................................................................................................. xv

About this information........................................................................................ xixPrerequisite and related information.................................................................................................... xxxivConventions used in this information....................................................................................................xxxivHow to send your comments..................................................................................................................xxxv

Summary of changes...................................................................................... xxxvii

Chapter 1. Configuring the GPFS cluster.................................................................1Creating your GPFS cluster..........................................................................................................................1Displaying cluster configuration information.............................................................................................. 2

Basic configuration information............................................................................................................. 3Information about protocol nodes.........................................................................................................3

Adding nodes to a GPFS cluster.................................................................................................................. 4Deleting nodes from a GPFS cluster............................................................................................................5Changing the GPFS cluster configuration data............................................................................................7Security mode............................................................................................................................................ 21

Setting security mode for internode communications in a cluster.....................................................23Minimum release level of a cluster............................................................................................................23Running IBM Spectrum Scale commands without remote root login......................................................26

Configuring sudo...................................................................................................................................27Configuring the cluster to use sudo wrapper scripts.......................................................................... 28Configuring IBM Spectrum Scale GUI to use sudo wrapper............................................................... 29Configuring a cluster to stop using sudo wrapper scripts...................................................................29Root-level processes that call administration commands directly.................................................... 29

Cluster quorum with quorum nodes......................................................................................................... 29Cluster quorum with quorum nodes and tiebreaker disks....................................................................... 30Displaying and changing the file system manager node.......................................................................... 32Starting and stopping GPFS.......................................................................................................................33Shutting down an IBM Spectrum Scale cluster........................................................................................ 35

Chapter 2. Configuring the CES and protocol configuration................................... 37Configuring Cluster Export Services ......................................................................................................... 37

Setting up Cluster Export Services shared root file system................................................................37Configuring Cluster Export Services nodes......................................................................................... 38Configuring CES protocol service IP addresses.................................................................................. 40CES IP aliasing to network adapters on protocol nodes..................................................................... 41Deploying Cluster Export Services packages on existing IBM Spectrum Scale 4.1.1 and later

nodes............................................................................................................................................... 45Verifying the final CES configurations..................................................................................................46

Creating and configuring file systems and filesets for exports................................................................ 47Configuring with the installation toolkit.................................................................................................... 47Deleting a Cluster Export Services node from an IBM Spectrum Scale cluster.......................................48Setting up Cluster Export Services groups in an IBM Spectrum Scale cluster........................................ 48

Chapter 3. Configuring and tuning your system for GPFS.......................................51General system configuration and tuning considerations........................................................................51

Clock synchronization.......................................................................................................................... 51

iii

GPFS administration security.............................................................................................................. 52Cache usage......................................................................................................................................... 52Access patterns....................................................................................................................................55Aggregate network interfaces..............................................................................................................55Swap space...........................................................................................................................................56

Linux configuration and tuning considerations.........................................................................................56updatedb considerations..................................................................................................................... 57Memory considerations........................................................................................................................57GPFS helper threads............................................................................................................................ 57Communications I/O............................................................................................................................ 57Disk I/O.................................................................................................................................................58

AIX configuration and tuning considerations............................................................................................58GPFS use with Oracle........................................................................................................................... 58

Chapter 4. Parameters for performance tuning and optimization...........................61Tuning parameters change history............................................................................................................63

Chapter 5. Ensuring high availability of the GUI service........................................ 69

Chapter 6. Configuring and tuning your system for Cloud services.........................71Configuration command execution matrix................................................................................................71Designating the Cloud services nodes...................................................................................................... 72Starting up the Cloud services software................................................................................................... 73Managing a cloud storage account............................................................................................................74

Amazon S3............................................................................................................................................74Swift3 account......................................................................................................................................75IBM Cloud Object Storage....................................................................................................................75Openstack Swift................................................................................................................................... 76Microsoft Azure.................................................................................................................................... 77

Defining cloud storage access points (CSAP)........................................................................................... 78Creating Cloud services............................................................................................................................. 79Configuring Cloud services with SKLM (optional)..................................................................................... 80Binding your file system or fileset to the Cloud service by creating a container pair set........................81Backing up the Cloud services database to the cloud.............................................................................. 84Backing up the Cloud services configuration............................................................................................84Configuring the maintenance windows.....................................................................................................85Enabling a policy for Cloud data sharing export service...........................................................................88Tuning Cloud services parameters............................................................................................................88Integrating Cloud services metrics with the performance monitoring tool.............................................91

GPFS-based configuration................................................................................................................... 91File-based configuration...................................................................................................................... 92

Setting up Transparent cloud tiering service on a remotely mounted client...........................................94Deploying WORM solutions....................................................................................................................... 95

Creating immutable filesets and files ................................................................................................. 95Setting up Transparent cloud tiering for WORM solutions..................................................................97

Chapter 7. Configuring the message queue.........................................................105Enabling the message queue.................................................................................................................. 105Disabling the message queue................................................................................................................. 105Disabling the message queue and removing the configuration............................................................. 105Actions that the mmmsgqueue command takes to enable the message queue.................................... 105Message queue limitations......................................................................................................................107

Chapter 8. Configuring file audit logging............................................................. 109Enabling file audit logging on a file system.............................................................................................109Disabling file audit logging on a file system............................................................................................109

iv

Actions that the mmaudit command takes to enable file audit logging................................................109Actions that the mmaudit command takes to disable file audit logging...............................................111Enabling and disabling file audit logging using the GUI......................................................................... 111Viewing file systems that have file audit logging enabled with the GUI................................................ 111Enabling file audit logging on an owning cluster for a file system that is remotely mounted...............112

Chapter 9. Configuring clustered watch folder.................................................... 113Enabling a clustered watch..................................................................................................................... 113Suspending a clustered watch................................................................................................................ 113Resuming a clustered watch................................................................................................................... 113Disabling a clustered watch.................................................................................................................... 114Configuration of an external Kafka sink in the IBM Spectrum Scale cluster......................................... 114Actions that the mmwatch command takes to enable a clustered watch..............................................115Actions that the mmwatch command takes to suspend a clustered watch...........................................116Actions that the mmwatch command takes to resume a clustered watch............................................ 116Actions that the mmwatch command takes to disable a clustered watch.............................................117Actions that the mmwatch command takes to auto-disable a clustered watch....................................118Actions that the mmwatch all upgrade command takes................................................................. 118

Chapter 10. Configuring Active File Management................................................119Configuration parameters for AFM..........................................................................................................119Parallel data transfer configuration parameters for AFM.......................................................................124Configuration changes in an existing AFM relationship..........................................................................126

Adding gateway nodes to the cache cluster......................................................................................126The NFS server at the home cluster.................................................................................................. 126

Chapter 11. Configuring AFM-based DR..............................................................129Configuration parameters for AFM-based DR.........................................................................................129Parallel data transfer configuration parameters for AFM-based DR..................................................... 131Changing configuration in an existing AFM DR relationship...................................................................132

Changing NFS server at secondary....................................................................................................133Changing gateway nodes in primary..................................................................................................133

Chapter 12. Tuning for Kernel NFS backend on AFM and AFM DR........................ 135Tuning the gateway node on the NFS client........................................................................................... 135Tuning on both the NFS client (gateway) and the NFS server (the home/secondary cluster)..............135Tuning the NFS server on the home/secondary cluster or the NFS server............................................136

Chapter 13. Performing GPFS administration tasks............................................ 139Requirements for administering a GPFS file system.............................................................................. 139

adminMode configuration attribute...................................................................................................140Common GPFS command principles...................................................................................................... 141

Specifying nodes as input to GPFS commands.................................................................................141Stanza files......................................................................................................................................... 142Listing active IBM Spectrum Scale commands................................................................................. 143

Determining how long mmrestripefs takes to complete....................................................................144

Chapter 14. Verifying network operation with the mmnetverify command........... 147

Chapter 15. Managing file systems..................................................................... 149Mounting a file system.............................................................................................................................149

Mounting a file system on multiple nodes.........................................................................................150Mount options specific to IBM Spectrum Scale................................................................................ 150Mounting a file system through GUI ................................................................................................. 151Changing a file system mount point on protocol nodes....................................................................152

Unmounting a file system........................................................................................................................153

v

Unmounting a file system on multiple nodes....................................................................................153Unmounting a file system through GUI ............................................................................................ 154

Deleting a file system.............................................................................................................................. 154Determining which nodes have a file system mounted..........................................................................155Checking and repairing a file system...................................................................................................... 155Dynamic validation of descriptors on disk.............................................................................................. 157File system maintenance mode ............................................................................................................. 157Listing file system attributes................................................................................................................... 160Modifying file system attributes..............................................................................................................161Querying and changing file replication attributes...................................................................................161

Querying file replication.....................................................................................................................161Changing file replication attributes................................................................................................... 162

Using Direct I/O on a file in a GPFS file system...................................................................................... 162File compression......................................................................................................................................163Setting the Quality of Service for I/O operations (QoS)..........................................................................169Restriping a GPFS file system................................................................................................................. 171Querying file system space..................................................................................................................... 173Querying and reducing file system fragmentation................................................................................. 174

Querying file system fragmentation.................................................................................................. 174Reducing file system fragmentation..................................................................................................175

Protecting data in a file system using backup........................................................................................ 176Protecting data in a file system using the mmbackup command.....................................................176Backing up a file system using the GPFS policy engine....................................................................182Backing up file system configuration information.............................................................................182Using APIs to develop backup applications...................................................................................... 183

Scale Out Backup and Restore (SOBAR).................................................................................................184Scheduling backups using IBM Spectrum Protect scheduler................................................................ 184Configuration reference for using IBM Spectrum Protect with IBM Spectrum Scale........................... 185

Options in the IBM Spectrum Protect configuration file dsm.sys.................................................... 185Options in the IBM Spectrum Protect configuration file dsm.opt.................................................... 187Base IBM Spectrum Protect client configuration files for IBM Spectrum Scale usage................... 189

Restoring a subset of files or directories from a local file system snapshot......................................... 190Restoring a subset of files or directories from a local fileset snapshot.................................................191Restoring a subset of files or directories from local snapshots using the sample script......................192Creating and managing file systems using GUI...................................................................................... 193

Chapter 16. File system format changes between versions of IBM SpectrumScale............................................................................................................. 199

Chapter 17. Managing disks............................................................................... 203Displaying disks in a GPFS cluster.......................................................................................................... 203Adding disks to a file system...................................................................................................................204Deleting disks from a file system............................................................................................................ 204Replacing disks in a GPFS file system.....................................................................................................206Additional considerations for managing disks........................................................................................207Displaying GPFS disk states.................................................................................................................... 208

Disk availability.................................................................................................................................. 208Disk status..........................................................................................................................................208

Changing GPFS disk states and parameters...........................................................................................209Changing your NSD configuration........................................................................................................... 211Changing NSD server usage and failback................................................................................................212Periodic check for NSD local disks..........................................................................................................212Enabling and disabling Persistent Reserve.............................................................................................212

Chapter 18. Managing protocol services............................................................. 215Configuring and enabling SMB and NFS protocol services.....................................................................215Support of vfs_fruit for the SMB protocol ........................................................................................... 216

vi

Configuring and enabling the Object protocol service........................................................................... 218Performance tuning for object services.............................................................................................218

Configuring and enabling the BLOCK service..........................................................................................219Disabling protocol services..................................................................................................................... 221

Chapter 19. Managing protocol user authentication............................................ 223Setting up authentication servers to configure protocol user access....................................................223

Integrating with AD server.................................................................................................................224Integrating with LDAP server.............................................................................................................225Integrating with Keystone Identity Service.......................................................................................230

Configuring authentication and ID mapping for file access................................................................... 230Prerequisite for configuring Kerberos-based SMB access............................................................... 232Configuring AD-based authentication for file access........................................................................233Configuring LDAP-based authentication for file access....................................................................243Configuring NIS-based authentication..............................................................................................248Authentication considerations for NFSv4 based access.................................................................. 249Prerequisites for configuring Kerberos based NFS access...............................................................250

Managing user-defined authentication...................................................................................................251Configuring authentication for object access......................................................................................... 256

Configuring local authentication for object access........................................................................... 257Configuring an AD-based authentication for object access..............................................................259Configuring an LDAP-based authentication for object access..........................................................262Configuring object authentication with an external keystone server...............................................265Creating object accounts................................................................................................................... 266Managing object users, roles, and projects.......................................................................................268Deleting expired tokens..................................................................................................................... 272

Deleting the authentication and the ID mapping configuration.............................................................273Listing the authentication configuration................................................................................................. 275Verifying the authentication services configured in the system............................................................ 276Modifying the authentication method .................................................................................................... 278Authentication limitations....................................................................................................................... 279

Chapter 20. Managing protocol data exports.......................................................283Managing SMB shares............................................................................................................................. 283

Creating SMB share............................................................................................................................283Changing SMB share configuration....................................................................................................284Creating SMB share ACLs...................................................................................................................284Removing SMB shares....................................................................................................................... 285Listing SMB shares............................................................................................................................. 285Managing SMB shares using MMC..................................................................................................... 285

Managing NFS exports.............................................................................................................................294Creating NFS exports......................................................................................................................... 294Changing NFS export configuration...................................................................................................295Removing NFS exports.......................................................................................................................295Listing NFS exports............................................................................................................................ 296GUI navigation for NFS exports......................................................................................................... 296Making bulk changes to NFS exports................................................................................................ 296

Multiprotocol exports.............................................................................................................................. 299Multiprotocol export considerations.......................................................................................................299

Chapter 21. Managing object storage..................................................................301Understanding and managing Object services....................................................................................... 301Understanding the mapping of OpenStack commands to IBM Spectrum Scale administrator

commands.......................................................................................................................................... 303Changing Object configuration values.................................................................................................... 304Changing the object base configuration to enable S3 API..................................................................... 304Configuring OpenStack EC2 credentials................................................................................................. 305

vii

Managing the OpenStack S3 API............................................................................................................ 305Managing object capabilities...................................................................................................................307Managing object versioning ....................................................................................................................307

Enabling object versioning.................................................................................................................307Disabling object versioning................................................................................................................ 308Creating a version of an object: Example.......................................................................................... 308

Mapping of storage policies to filesets................................................................................................... 310Administering storage policies for object storage..................................................................................310

Creating storage policy for object compression................................................................................311Creating storage policy for object encryption................................................................................... 312

Adding a region in a multi-region object deployment............................................................................ 313Administering a multi-region object deployment environment............................................................. 314Unified file and object access in IBM Spectrum Scale .......................................................................... 315

Enabling object access to existing filesets........................................................................................315Identity management modes for unified file and object access...................................................... 317Authentication in unified file and object access............................................................................... 322Validating shared authentication ID mapping...................................................................................322The objectizer process.......................................................................................................................324File path in unified file and object access......................................................................................... 325Administering unified file and object access.....................................................................................326In-place analytics using unified file and object access.................................................................... 339Limitations of unified file and object access..................................................................................... 340Constraints applicable to unified file and object access...................................................................341Data ingestion examples....................................................................................................................342curl commands for unified file and object access related user tasks.............................................. 343

Configuration files for IBM Spectrum Scale for object storage..............................................................344Backing up and restoring object storage................................................................................................ 347

Backing up the object storage........................................................................................................... 348Restoring the object storage..............................................................................................................350

Configuration of object for isolated node and network groups..............................................................352Enabling the object heatmap policy........................................................................................................354

Chapter 22. Managing GPFS quotas....................................................................357Enabling and disabling GPFS quota management..................................................................................357Default quotas......................................................................................................................................... 359Implications of quotas for different protocols........................................................................................362Explicitly establishing and changing quotas...........................................................................................363Setting quotas for users on a per-project basis......................................................................................365Checking quotas...................................................................................................................................... 368Listing quotas...........................................................................................................................................370Activating quota limit checking............................................................................................................... 371Deactivating quota limit checking........................................................................................................... 373Changing the scope of quota limit checking........................................................................................... 374Creating file system quota reports..........................................................................................................375Restoring quota files................................................................................................................................376

Chapter 23. Managing GUI users........................................................................ 379

Chapter 24. Managing GPFS access control lists................................................. 385Traditional GPFS ACL administration......................................................................................................385

Setting traditional GPFS access control lists.................................................................................... 386Displaying traditional GPFS access control lists............................................................................... 387Applying an existing traditional GPFS access control list.................................................................388Changing traditional GPFS access control lists.................................................................................388Deleting traditional GPFS access control lists.................................................................................. 389

NFS V4 ACL administration..................................................................................................................... 389NFS V4 ACL Syntax.............................................................................................................................390

viii

NFS V4 ACL translation...................................................................................................................... 392Setting NFS V4 access control lists................................................................................................... 393Displaying NFS V4 access control lists..............................................................................................394Applying an existing NFS V4 access control list................................................................................394Changing NFS V4 access control lists............................................................................................... 395Deleting NFS V4 access control lists................................................................................................. 395Considerations when using GPFS with NFS V4 ACLs........................................................................395

Authorizing protocol users...................................................................................................................... 396Authorizing file protocol users...........................................................................................................396Authorizing object users.................................................................................................................... 409Authorization limitations....................................................................................................................415

Chapter 25. Native NFS and GPFS...................................................................... 417Exporting a GPFS file system using NFS................................................................................................. 417

Export considerations........................................................................................................................ 418NFS usage of GPFS cache........................................................................................................................420Synchronous writing using NFS...............................................................................................................421Unmounting a file system after NFS export............................................................................................421NFS automount considerations...............................................................................................................421Clustered NFS and GPFS on Linux.......................................................................................................... 422

Chapter 26. Considerations for GPFS applications.............................................. 423Exceptions to Open Group technical standards..................................................................................... 423Determining if a file system is controlled by GPFS.................................................................................423Exceptions and limitations to NFS V4 ACLs support.............................................................................. 424

Linux ACLs and extended attributes..................................................................................................424General CES NFS Linux limitations..........................................................................................................425Considerations for the use of direct I/O (O_DIRECT).............................................................................425

Chapter 27. Accessing a remote GPFS file system...............................................427Remote user access to a GPFS file system.............................................................................................429Using NFS/SMB protocol over remote cluster mounts...........................................................................430

Configuring protocols on a separate cluster..................................................................................... 431Managing multi-cluster protocol environments................................................................................431Upgrading multi-cluster environments............................................................................................. 432Limitations of protocols on remotely mounted file systems............................................................ 432

Mounting a remote GPFS file system......................................................................................................433Managing remote access to a GPFS file system..................................................................................... 435Using remote access with multiple network definitions........................................................................ 436Using multiple security levels for remote access................................................................................... 438Changing security keys with remote access........................................................................................... 439NIST compliance..................................................................................................................................... 440Important information about remote access......................................................................................... 441

Chapter 28. Information lifecycle management for IBM Spectrum Scale..............443Storage pools........................................................................................................................................... 443

Internal storage pools........................................................................................................................444External storage pools....................................................................................................................... 448

Policies for automating file management............................................................................................... 449Overview of policies........................................................................................................................... 449Policy rules......................................................................................................................................... 451The mmapplypolicy command and policy rules................................................................................471Policy rules: Examples and tips......................................................................................................... 475Managing policies...............................................................................................................................481Working with external storage pools.................................................................................................488Backup and restore with storage pools.............................................................................................493ILM for snapshots.............................................................................................................................. 495

ix

User storage pools...................................................................................................................................497File heat: Tracking file access temperature............................................................................................497Filesets.....................................................................................................................................................499

Fileset namespace............................................................................................................................. 500Filesets and quotas............................................................................................................................ 501Filesets and storage pools................................................................................................................. 502Filesets and global snapshots........................................................................................................... 502Fileset-level snapshots......................................................................................................................503Filesets and backup........................................................................................................................... 504Managing filesets............................................................................................................................... 505

Immutability and appendOnly features..................................................................................................510

Chapter 29. Creating and maintaining snapshots of file systems......................... 515Creating a snapshot.................................................................................................................................515Listing snapshots..................................................................................................................................... 517Restoring a file system from a snapshot.................................................................................................517Reading a snapshot with the policy engine............................................................................................ 519Linking to a snapshot...............................................................................................................................519Deleting a snapshot................................................................................................................................. 521Managing snapshots using IBM Spectrum Scale GUI............................................................................ 521

Chapter 30. Creating and managing file clones................................................... 525Creating file clones.................................................................................................................................. 525Listing file clones..................................................................................................................................... 526Deleting file clones.................................................................................................................................. 527Splitting file clones from clone parents.................................................................................................. 527File clones and disk space management................................................................................................ 527File clones and snapshots....................................................................................................................... 527File clones and policy files...................................................................................................................... 528

Chapter 31. Scale Out Backup and Restore (SOBAR)........................................... 529Backup procedure with SOBAR...............................................................................................................529Restore procedure with SOBAR.............................................................................................................. 531

Chapter 32. Data Mirroring and Replication........................................................ 535General considerations for using storage replication with GPFS...........................................................536Data integrity and the use of consistency groups...................................................................................536Handling multiple versions of IBM Spectrum Scale data.......................................................................536Continuous Replication of IBM Spectrum Scale data.............................................................................537

Synchronous mirroring with GPFS replication.................................................................................. 537Synchronous mirroring utilizing storage based replication.............................................................. 548Point-in-time copy of IBM Spectrum Scale data...............................................................................556

Chapter 33. Implementing a clustered NFS environment on Linux.......................559NFS monitoring........................................................................................................................................ 559NFS failover..............................................................................................................................................560NFS locking and load balancing.............................................................................................................. 560CNFS network setup................................................................................................................................ 561CNFS setup.............................................................................................................................................. 561CNFS administration................................................................................................................................563

Chapter 34. Implementing Cluster Export Services.............................................565CES features............................................................................................................................................ 565

CES cluster setup............................................................................................................................... 565CES network configuration................................................................................................................ 566CES address failover and distribution policies..................................................................................567

x

CES protocol management................................................................................................................ 569CES management and administration............................................................................................... 569

CES NFS support......................................................................................................................................569CES SMB support..................................................................................................................................... 572CES OBJ support......................................................................................................................................573CES HDFS support................................................................................................................................... 575Migration of CNFS clusters to CES clusters............................................................................................ 576

Chapter 35. Identity management on Windows / RFC 2307 Attributes................ 579Auto-generated ID mappings..................................................................................................................579Configuring ID mappings in Active Directory Users and Computers for Windows Server 2016 (and

subsequent) versions......................................................................................................................... 580Installing Windows IDMU........................................................................................................................583Configuring ID mappings in IDMU...........................................................................................................584

Chapter 36. Protocols cluster disaster recovery.................................................. 587Protocols cluster disaster recovery limitations and prerequisites........................................................ 587Example setup for protocols disaster recovery...................................................................................... 588Setting up gateway nodes to ensure cluster communication during failover........................................589Creating the inband disaster recovery setup..........................................................................................589Creating the outband disaster recovery setup....................................................................................... 591Performing failover for protocols cluster when primary cluster fails.....................................................593

Re-create file export configuration....................................................................................................593Restore file export configuration....................................................................................................... 593

Performing failback to old primary for protocols cluster....................................................................... 594Re-create file protocol configuration for old primary....................................................................... 594Restore file protocol configuration for old primary...........................................................................595

Performing failback to new primary for protocols cluster......................................................................597Re-create file protocol configuration for new primary......................................................................597Restore file protocol configuration for new primary......................................................................... 600

Backing up and restoring protocols and CES configuration information............................................... 604Updating protocols and CES configuration information......................................................................... 605Protocols and cluster configuration data required for disaster recovery.............................................. 605

Object data required for protocols cluster DR.................................................................................. 605SMB data required for protocols cluster DR......................................................................................612NFS data required for protocols cluster DR...................................................................................... 614Authentication related data required for protocols cluster DR........................................................ 615CES data required for protocols cluster DR.......................................................................................617

Chapter 37. File Placement Optimizer................................................................ 619Distributing data across a cluster........................................................................................................... 623FPO pool file placement and AFM...........................................................................................................623Configuring FPO....................................................................................................................................... 624

Configuring IBM Spectrum Scale Clusters........................................................................................ 624Basic Configuration Recommendations............................................................................................ 629Configuration and tuning of Hadoop workloads................................................................................641Configuration and tuning of database workloads............................................................................. 641Configuring and tuning SparkWorkloads........................................................................................... 642

Ingesting data into IBM Spectrum Scale clusters.................................................................................. 642Exporting data out of IBM Spectrum Scale clusters...............................................................................643Upgrading FPO.........................................................................................................................................643Monitoring and administering IBM Spectrum Scale FPO clusters......................................................... 646

Rolling upgrades................................................................................................................................ 646The IBM Spectrum Scale FPO cluster............................................................................................... 648Failure detection................................................................................................................................ 650Disk Failures....................................................................................................................................... 651Node failure........................................................................................................................................653

xi

Handling multiple nodes failure.........................................................................................................655Network switch failure....................................................................................................................... 656Data locality........................................................................................................................................656Disk Replacement.............................................................................................................................. 664

Auto recovery...........................................................................................................................................666Failure and recovery...........................................................................................................................667QoS support for autorecovery............................................................................................................669

Restrictions..............................................................................................................................................669

Chapter 38. Encryption...................................................................................... 671Encryption keys....................................................................................................................................... 671Encryption policies.................................................................................................................................. 672Encryption policy rules............................................................................................................................ 672Preparation for encryption...................................................................................................................... 677Establishing an encryption-enabled environment..................................................................................683

Simplified setup: Using SKLM with a self-signed certificate............................................................ 683Simplified setup: Using SKLM with a certificate chain...................................................................... 691Simplified setup: Valid and invalid configurations............................................................................ 700Simplified setup: Accessing a remote file system............................................................................ 703Simplified setup: Doing other tasks...................................................................................................707Regular setup: Using SKLM with a self-signed certificate................................................................ 713Regular setup: Using SKLM with a certificate chain..........................................................................721Configuring encryption with SKLM v2.7 or later................................................................................730Configuring encryption with the Vormetric DSM key server............................................................. 733

Certificate expiration warnings............................................................................................................... 740Renewing client and server certificates..................................................................................................743

Certificate expiration errors...............................................................................................................744Renewing expired server certificates................................................................................................ 744Renewing expired client certificates................................................................................................. 750

Encryption hints.......................................................................................................................................755Secure deletion........................................................................................................................................756Key rotation: Replacing master encryption keys....................................................................................758Encryption and standards compliance....................................................................................................760

Encryption and FIPS 140-2 certification...........................................................................................760Encryption and NIST SP800-131A compliance................................................................................761

Encryption in a multicluster environment...............................................................................................761Encryption in a Disaster Recovery environment.....................................................................................761Encryption and backup/restore...............................................................................................................761Encryption and snapshots....................................................................................................................... 761Encryption and a local read-only cache (LROC) device..........................................................................762Encryption and external pools................................................................................................................ 762Encryption requirements and limitations............................................................................................... 763

Chapter 39. Managing certificates to secure communications between GUI webserver and web browsers............................................................................... 765

Chapter 40. Securing protocol data.................................................................... 767Planning for protocol data security......................................................................................................... 769Configuring protocol data security..........................................................................................................769

Enabling secured connection between the IBM Spectrum Scale system and authenticationserver.............................................................................................................................................770

Securing data transfer........................................................................................................................773Securing NFS data transfer................................................................................................................ 773Securing SMB data transfer............................................................................................................... 775Secured object data transfer............................................................................................................. 775

Data security limitations..........................................................................................................................775

xii

Chapter 41. Cloud services: Transparent cloud tiering and Cloud data sharing.....777Administering files for Transparent cloud tiering................................................................................... 777

Applying a policy on a Transparent cloud tiering node.....................................................................777Migrating files to the cloud storage tier.............................................................................................780Pre-migrating files to the cloud storage tier......................................................................................780Recalling files from the cloud storage tier.........................................................................................782Reconciling files between IBM Spectrum Scale file system and cloud storage tier........................ 782Cleaning up files transferred to the cloud storage tier..................................................................... 783Deleting cloud objects....................................................................................................................... 784Managing reversioned files................................................................................................................ 785Listing files migrated to the cloud storage tier..................................................................................785Restoring files.....................................................................................................................................786Restoring Cloud services configuration.............................................................................................787Checking the Cloud services database integrity............................................................................... 788Manual recovery of Transparent cloud tiering database.................................................................. 788Scale out backup and restore (SOBAR) for Cloud services...............................................................789

Cloud data sharing...................................................................................................................................802Listing files exported to the cloud..................................................................................................... 803Importing cloud objects exported through an old version of Cloud data sharing............................805

Administering Transparent cloud tiering and Cloud data sharing services........................................... 805Stopping Cloud services software..................................................................................................... 806Monitoring the health of Cloud services software.............................................................................806Checking the Cloud services version................................................................................................. 807

Known limitations of Cloud services ...................................................................................................... 808

Chapter 42. Managing file audit logging.............................................................. 811Starting consumers in file audit logging..................................................................................................811Stopping consumers in file audit logging................................................................................................ 811Managing the list of monitored events....................................................................................................811Designating additional broker nodes for increased performance..........................................................812

Chapter 43. Performing a watch with the watch folder API................................. 813

Chapter 44. RDMA tuning...................................................................................815

Chapter 45. Administering AFM..........................................................................817Creating an AFM relationship by using the NFS protocol....................................................................... 817

Setting up the home cluster...............................................................................................................817Setting up the cache cluster.............................................................................................................. 818Example of creating an AFM relationship by using the NFS protocol ..............................................819Example of AFM support for Kerberos-enabled NFS protocol exports............................................821

Creating an AFM relationship by using GPFS protocol........................................................................... 822Setting up the home cluster...............................................................................................................822Setting up the cache cluster.............................................................................................................. 822Example of creating an AFM relationship by using the GPFS protocol............................................ 823

Chapter 46. Administering AFM DR.................................................................... 825Creating an AFM-based DR relationship.................................................................................................825Converting GPFS filesets to AFM DR....................................................................................................... 826Converting AFM relationship to AFM DR.................................................................................................827

Chapter 47. Highly available write cache (HAWC)............................................... 829Applications that can benefit from HAWC.............................................................................................. 829Restrictions and tuning recommendations for HAWC............................................................................830Using HAWC............................................................................................................................................. 830

xiii

Chapter 48. Local read-only cache..................................................................... 833

Chapter 49. Miscellaneous advanced administration topics................................ 835Changing IP addresses and host names.................................................................................................835Enabling a cluster for IPv6...................................................................................................................... 836Using multiple token servers...................................................................................................................837Exporting file system definitions between clusters................................................................................837IBM Spectrum Scale port usage..............................................................................................................838Securing the IBM Spectrum Scale system using firewall.......................................................................840

Firewall recommendations for the IBM Spectrum Scale installation...............................................841Firewall recommendations for internal communication among nodes........................................... 842Firewall recommendations for protocol access................................................................................843Firewall recommendations for IBM Spectrum Scale GUI.................................................................848Firewall recommendations for IBM SKLM.........................................................................................849Firewall recommendations for Vormetric DSM................................................................................. 850Firewall recommendations for the REST API....................................................................................850Firewall recommendations for Performance Monitoring tool...........................................................850Firewall considerations for Active File Management (AFM)............................................................. 852Firewall considerations for remote mounting of file systems.......................................................... 852Firewall recommendations for using IBM Spectrum Protect with IBM Spectrum Scale.................852Firewall considerations for using IBM Spectrum Archive with IBM Spectrum Scale...................... 853Firewall recommendations for file audit logging, watch folder API, and clustered watch folder... 853Firewall recommendations for call home..........................................................................................853Examples of how to open firewall ports............................................................................................ 854

Supported web browser versions and web browser settings for GUI................................................... 856

Chapter 50. GUI limitations................................................................................859

Accessibility features for IBM Spectrum Scale....................................................861Accessibility features.............................................................................................................................. 861Keyboard navigation................................................................................................................................861IBM and accessibility...............................................................................................................................861

Notices..............................................................................................................863Trademarks..............................................................................................................................................864Terms and conditions for product documentation.................................................................................864IBM Online Privacy Statement................................................................................................................ 865

Glossary............................................................................................................ 867

Index................................................................................................................ 875

xiv

Tables

1. IBM Spectrum Scale library information units............................................................................................xx

2. Conventions..............................................................................................................................................xxxv

3. Features stabilized in Version 5.0.4.......................................................................................................... xliv

4. Features deprecated in Version 5.0.4........................................................................................................ xlv

5. Features discontinued in Version 5.0.4.....................................................................................................xlvi

6. List of changes in documentation.............................................................................................................xlvii

7. Configuration attributes on the mmchconfig command.............................................................................. 9

8. Attributes and default values......................................................................................................................88

9. Supported Components.............................................................................................................................. 89

10. AFM configuration parameters used at a home or cache cluster..........................................................119

11. Configuration parameters at cache for parallel data transfer............................................................... 124

12. AFM DR configuration parameters used at a primary or secondary cluster......................................... 129

13. Configuration parameters at cache for parallel data transfer............................................................... 131

14. Configuration parameters at cache for parallel data transfer - valid values.........................................132

15. NFS server parameters........................................................................................................................... 136

16. Compression libraries and their required file system format level and format number...................... 163

17. COMPRESSION and illCompressed flags............................................................................................... 167

18. Set QoS classes to unlimited.................................................................................................................. 170

19. Allocate the available IOPS.................................................................................................................... 171

20. Authentication requirements for each file access protocol. .................................................................253

21. Object services and protocol nodes....................................................................................................... 302

22. Object input behavior in unified_mode..................................................................................................320

23. Configuration options for [swift-constraints] in swift.conf....................................................................341

xv

24. Configurable options for [DEFAULT] in object-server-sof.conf............................................................. 344

25. Configurable options for [capabilities] in spectrum-scale-object.conf.................................................345

26. Configuration options for [DEFAULT] in spectrum-scale-objectizer.conf............................................. 346

27. Configuration options for [IBMOBJECTIZER-LOGGER] in spectrum-scale-objectizer.conf................ 346

28. Configuration options for object-server.conf.........................................................................................346

29. Configuration options for /etc/sysconfig/memcached.......................................................................... 347

30. Configuration options for proxy-server.conf..........................................................................................347

31. mkldap command parameters............................................................................................................... 381

32. Removal of a file with ACL entries DELETE and DELETE_CHILD........................................................... 392

33. Mapping from NFSv4 ACL entry to SMB Security Descriptor.................................................................398

34. Mapping from SMB Security Descriptor to NFSv4 ACL entry with unixmap or ldapmap id mapping...398

35. Mapping from SMB Security Descriptor to NFSv4 ACL entry with default id mapping.........................399

36. ACL permissions required to work on files and directories, while using SMB protocol (table 1 of 2). 403

37. ACL permissions required to work on files and directories, while using SMB protocol (table 2 of 2). 403

38. ACL permissions required to work on files and directories, while using NFS protocol (table 1 of 2).. 404

39. ACL permissions required to work on files and directories, while using NFS protocol (table 2 of 2).. 405

40. Commands and reference to manage ACL tasks................................................................................... 408

41. ACL options that are available to manipulate object read ACLs............................................................413

42. Summary of commands to set up cross-cluster file system access..................................................... 435

43. Effects of options on uncompressed or compressed files.................................................................... 454

44. The effects of file operations on an immutable file or an appendOnly file........................................... 511

45. IAM modes and their effects on file operations on immutable files..................................................... 512

46. Example for retention period..................................................................................................................522

47. Example - Time stamp of snapshots that are retained based on the retention policy.........................522

48. User identification attributes..................................................................................................................582

xvi

49. Group identification attribute................................................................................................................. 582

50. Valid EncParamString values..................................................................................................................673

51. Valid combine parameter string values..................................................................................................673

52. Valid wrapping parameter string values.................................................................................................673

53. Required version of IBM Spectrum Scale...................................................