parallel file system feature guide - huawei cloud · object storage service parallel file system...

Object Storage Service

Parallel File System Feature Guide

Issue 02

Date 2019-08-15

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Issue 02 (2019-08-15) Copyright © Huawei Technologies Co., Ltd. i

Contents

1 Introduction.............................................................................................................................. 11.1 About Parallel File System................................................................................................................................................... 11.2 Application Scenarios............................................................................................................................................................. 11.3 Constraints................................................................................................................................................................................. 21.4 Using PFS................................................................................................................................................................................... 21.5 Pricing.......................................................................................................................................................................................... 3

2 Managing Parallel File Systems on OBS Console............................................................ 42.1 Creating a Parallel File System........................................................................................................................................... 4

3 Managing Parallel File Systems by APIs............................................................................ 63.1 Supported APIs......................................................................................................................................................................... 6

4 Managing Parallel File Systems by obsfs........................................................................ 13

A Change History...................................................................................................................... 14

Object Storage ServiceParallel File System Feature Guide Contents

Issue 02 (2019-08-15) Copyright © Huawei Technologies Co., Ltd. ii

1 Introduction

1.1 About Parallel File System

1.2 Application Scenarios

1.3 Constraints

1.4 Using PFS

1.5 Pricing

1.1 About Parallel File SystemParallel File System (PFS) is a product of Object Storage Service (OBS). A parallelfile system is an optimized file system that delivers high performance. It providesmillisecond-level access latency, TB/s-level bandwidth, and million-level IOPS.Parallel file systems can quickly process HPC workloads.

You can access data in a parallel file system using the standard OBS APIs. You canalso use the tool obsfs (supporting POSIX) to mount parallel file systems to yourLinux cloud servers, so that you can process files and directories in the parallel filesystems online similar to the management of local file systems, including creating,deleting, renaming files and folders, as well as modifying files.

1.2 Application ScenariosPFS is highly compatible, scalable, and reliable, with amazing performancecapabilities. PFS is optimal for HPC and media asset archiving scenarios.

The main application scenarios of PFS are as follows:

Video surveillance: public security, business environments, and households

VOD: OTT delivery and media assets library

HPC: DNA sequencing and computer-aided engineering (CAE) scenarios inmanufacturing

Big Data: automobile industry, particularly, the Internet of Vehicles (IoV) analyticsscenarios

Object Storage ServiceParallel File System Feature Guide 1 Introduction

Issue 02 (2019-08-15) Copyright © Huawei Technologies Co., Ltd. 1

1.3 ConstraintsOperation Limitations

● An existing OBS bucket cannot be converted to a parallel file system. To use aparallel file system, you need to create it on OBS Console. For details abouthow to create a parallel file system, see 2.1 Creating a Parallel File System.

● A parallel file system can be created only on OBS Console.

Function Restrictions

● Image processing is not supported.

● Server-side encryption is not supported.

● Cross-region replication is not supported.

● Versioning is not supported.

● Bucket inventory is not supported.

● Lifecycle management is not supported.

● CORS is not supported.

● Static website hosting is not supported.

● Changing file storage class is not supported.

● Configuration of default storage class for a parallel file system is notsupported.

● Configuration for a single resource in a parallel file system is not supported.You can only configure a policy for a parallel file system as a whole.

● Dedicated OBS is not supported.

1.4 Using PFSPFS provides the console, client (obsfs), and REST APIs for managing andaccessing data. obsfs supports POSIX, so that your applications can be seamlesslyinterconnected with PFS, requiring no modification to your applications. You canprocess files stored in PFS anytime, anywhere, and quickly obtain processed files.PFS supports both POSIX and OBS APIs, therefore, you can still process files in theway that you process objects, implementing flexible conversion between objectsand files.

PFS provides the console and REST APIs for managing and accessing data. Yourapplications can be seamlessly interconnected with PFS, without making anymodification to your applications. You can process files stored in PFS anytime,anywhere, and quickly obtain processed files. PFS supports both POSIX and OBSAPIs, therefore, you can still process files in the way that you process objects,implementing flexible conversion between objects and files.

You can use PFS in the following ways:



Table 1-1 How to use PFS

Method Function Reference

Console On the console, you can createparallel file systems, and youcan also view and manage yourfile systems and files.

2.1 Creating a ParallelFile System

obsfs obsfs is a client developed anddesigned for PFS. It supportsPOSIX and can mount parallelfile systems to cloud servers.obsfs is recommended foraccessing PFS.

Introduction to obsfs

OBS API Use parallel file systems bycalling OBS APIs.

3.1 Supported APIs

Other OBS tools PFS allows conversions betweenobjects and files. Therefore, youcan also use other OBS tools(such as obsutil and OBSBrowser) to access PFS.However, these tools are notfully compatible with PFS yet,and further development isongoing. For bettercompatibility, obsfs isrecommended.

OBS Tools

1.5 PricingFor details about PFS pricing, see Product Pricing DetailsProduct Pricing Details.



https://support.huaweicloud.com/en-us/fstg-obs/obs_12_0001.html

https://support.huaweicloud.com/en-us/tg-obs/obs_09_0001.html

https://www.huaweicloud.com/en-us/pricing.html?tab=detail#/obs

2 Managing Parallel File Systems on OBSConsole

2.1 Creating a Parallel File System

2.1 Creating a Parallel File SystemYou can create a parallel file system on the console.

Procedure

Step 1 On the homepage of the console, click Service List on the top navigation menu,and choose Storage > Object Storage Service.

Step 2 In the navigation pane on the left, select Parallel File System.

Step 3 Click Create Parallel File System in the upper right corner of the page. The pagefor creating a parallel file system is displayed.

Object Storage ServiceParallel File System Feature Guide 2 Managing Parallel File Systems on OBS Console


Figure 2-1 Creating a parallel file system

Step 4 Select a region and enter a name for the parallel file system.

● Once a parallel file system is created, its name cannot be changed. Plan a proper namein advance.

● When a URL is used to access the parallel file system, the parallel file system name willbecome a part of the URL. According to DNS rules, URLs do not support uppercaseletters and cannot distinguish names containing uppercase letters. Therefore, a parallelfile system name can contain only lowercase letters, digits, periods (.), and hyphens (-).For example, if you attempt to access the parallel file system MyFileSystem using aURL, the file system name will be resolved to myfilesystem, causing an access error.

Step 5 Policy: You can select the Private, Public Read, or Public Read and Write policyfor the parallel file system.

Step 6 Tags: Tags are used to identify parallel file systems in OBS, for the purpose ofclassification. Each tag is represented by one key-value pair. For details about howto add a tag, see Tag Overviewthe section of "Tag Overview" in OBS User Guide.

Step 7 Confirm the price at the bottom of the page and click Create Now.

Step 8 Confirm the price at the bottom of the page and click Create Now.

Step 9 After the parallel file system is created, you can view it on the parallel file systemlist.

Then, you can use the parallel file system the way as you using a bucket. Fordetails about how to use PFS, see 1.4 Using PFS.

----End

Object Storage ServiceParallel File System Feature Guide 2 Managing Parallel File Systems on OBS Console


https://support.huaweicloud.com/en-us/usermanual-obs/en-us_topic_0059888284.html

3 Managing Parallel File Systems by APIs

3.1 Supported APIs

3.1 Supported APIsThis section focuses on the compatibility of parallel file systems and existing OBSAPIs, and provides a differential description of APIs that are partly compatible.

For details about the OBS APIs, see OBS API ReferenceOBS API Reference.

API Operations on Buckets

Table 3-1 API operations on buckets

API PFSCompatible

Differences

Listing buckets Yes The x-obs-bucket-type:POSIX header fieldis required for obtaining the list of parallelfile systems.

Creating abucket

Yes The x-obs-fs-file-interface:Enabledheader field is required for creating aparallel file system.

Listing objects ina bucket

Yes -

Obtaining bucketmetadata

Yes -

Obtaining bucketregion locations

Yes -

Deleting abucket

Yes -

Object Storage ServiceParallel File System Feature Guide 3 Managing Parallel File Systems by APIs


https://support.huaweicloud.com/en-us/api-obs/obs_04_0005.html

API Operations on Advanced Bucket Settings

Table 3-2 API operations on advanced bucket settings

API PFSCompatible

Differences

Configuring abucket policy

Yes Only bucket policies whose Resource is *are supported.

Obtaining bucketpolicyinformation

Yes -

Deleting abucket policy

Yes -

Configuring abucket ACL

Yes -

Obtaining bucketACL information

Yes -

Configuringlogging for abucket

Yes -

Obtaining abucket loggingconfiguration

Yes -

Configuringbucket lifecyclerules

No -

Obtaining bucketlifecycleconfiguration

No -

Deleting bucketlifecycle rules

No -

Configuringversioning for abucket

No -

Obtaining bucketversioning status

No -

Configuringeventnotification for abucket

Yes -



API PFSCompatible

Differences

Obtaining theeventnotificationconfiguration ofa bucket

Yes -

Configuringstorage class fora bucket

No -

Obtaining bucketstorage classinformation

No -

Configuringcross-regionreplication for abucket

No -

Obtaining thecross-regionreplicationconfiguration ofa bucket

No -

Deleting thecross-regionreplicationconfiguration ofa bucket

No -

Configuring tagsfor a bucket

Yes -

Obtaining buckettags

Yes -

Deleting buckettags

Yes -

Configuringbucket storagequota

Yes -

Querying bucketstorage quota

Yes -

Queryinginformationabout used spacein a bucket

Yes -



API PFSCompatible

Differences

Configuringbucketinventories

No -

Obtaining bucketinventories

No -

Listing bucketinventories

No -

Deleting bucketinventories

No -

Configuring acustom domainname for abucket

Yes -

Obtaining thecustom domainname of abucket

Yes -

Deleting acustom domainname of abucket

Yes -

Configuringbucketencryption

No -

Obtaining bucketencryptionconfiguration

No -

Deleting theencryptionconfiguration ofa bucket

No -

Setting the directreading policy forArchiveColdobjects in abucket

√ -

Obtaining thedirect readingpolicy forArchiveColdobjects in abucket

√ -



API PFSCompatible

Differences

Deleting thedirect readingpolicy forArchiveColdobjects in abucket

√ -

API Operations for Static Website Hosting

Table 3-3 API operations for static website hosting

API PFSCompatible

Differences

Configuring staticwebsite hostingfor a bucket

No -

Obtaining thestatic websitehostingconfiguration ofa bucket

No -

Deleting thestatic websitehostingconfiguration ofa bucket

No -

Configuringbucket CORS

No -

Obtaining theCORSconfiguration ofa bucket

No -

Deleting theCORSconfiguration ofa bucket

No -

OPTIONS buckets No -

OPTIONS objects No -



API Operations on Objects

Table 3-4 API operations on objects

API PFSCompatible

Differences

PUT objects Yes The following header fields are notsupported: x-obs-storage-class, x-obs-website-redirect-location, x-obs-server-side-encryption, x-obs-server-side-encryption-kms-key-id, x-obs-server-side-encryption-customer-algorithm, x-obs-server-side-encryption-customer-key, x-obs-server-side-encryption-customer-key-MD5, success-action-redirect, and x-obs-expires

POST objects Yes The following header fields are notsupported: x-obs-storage-class, x-obs-website-redirect-location, x-obs-server-side-encryption, x-obs-server-side-encryption-kms-key-id, x-obs-server-side-encryption-customer-algorithm, x-obs-server-side-encryption-customer-key, x-obs-server-side-encryption-customer-key-MD5, success-action-redirect, and x-obs-expires

Copying objects Yes Files in a parallel file system cannot becopied to a bucket. They can be copiedonly between parallel file systems.

Obtaining objectcontent

Yes -

Obtaining objectmetadata

Yes -

Deleting objects Yes -

Batch deletingobjects

Yes -

RestoringArchiveColdobjects

No -

Appendingobjects

No -

Configuringobject ACL

Yes -

Obtaining objectACL information

Yes -



API PFSCompatible

Differences

Modifying objectmetadata

Yes -

Modifying anobject

Yes This is a PFS only API, and is not supportedby OBS buckets.

Truncating anobject


Renaming anobject


API Operations for Multipart Tasks

Table 3-5 API operations for multipart tasks

API PFSCompatible

Differences

Listing initializedmultipart tasks ina bucket

Yes -

Initiatingmultipart uploadtasks

Yes -

Uploading parts Yes -

Copying parts Yes Copying parts is not supported for anappended file.

Listing uploadedparts

Yes -

Merging parts Yes -

Cancelingmultipart tasks

Yes -



4 Managing Parallel File Systems by obsfs

obsfs, built on the basis of Filesystem in Userspace (FUSE) and compatible withPOSIX, is a file system tool provided by Object Storage Service (OBS) for mountingOBS parallel file systems to Linux operating systems. It enables you to easilyaccess the infinite storage space on OBS in the same way as you operate the localfile system.

obsfs is an optimal option, if you are used to storing data in local file systems anddo not want to change the way of data accessing even though data is stored inOBS.

For more information about the obsfs tool, see Introduction to obsfs.

Object Storage ServiceParallel File System Feature Guide 4 Managing Parallel File Systems by obsfs


https://support.huaweicloud.com/en-us/fstg-obs/obs_12_0001.html

A Change History

ReleaseDate

What's New

2020-03-09 This is the first official release.

ReleaseDate

What's New

2019-11-30 This is the fourth official release.This issue incorporates the following change:● Added the lifecycle management feature.

2019-11-08 This is the third official release.This issue incorporates the following change:● Adjusted the billing mode of PFS.

2019-08-15 This is the second official release.This issue incorporates the following changes:● Added the performance restrictions of parallel file systems.● Added the list of APIs supported by parallel file systems.

2019-07-15 This is the first official release.

Object Storage ServiceParallel File System Feature Guide A Change History


parallel file system feature guide - huawei cloud · object storage service parallel file system...

Documents