1 ggf- grid storage management wg global grid forum grid storage management working group chairs:...

38
1 GGF- Grid Storage Management WG Global Grid Forum Global Grid Forum Grid Storage Management Grid Storage Management Working Group Working Group Chairs Chairs : Arie : Arie Shoshani (LBNL) Shoshani (LBNL) Peter Kunszt (CERN) Peter Kunszt (CERN) Secretary Secretary : Alex Sim : Alex Sim (LBNL) (LBNL) http://sdm.lbl.gov/gsm http://sdm.lbl.gov/gsm

Post on 21-Dec-2015

224 views

Category:

Documents


1 download

TRANSCRIPT

1 GGF- Grid Storage Management WG

Global Grid ForumGlobal Grid ForumGrid Storage ManagementGrid Storage Management

Working GroupWorking Group

ChairsChairs: Arie Shoshani (LBNL): Arie Shoshani (LBNL)

Peter Kunszt (CERN)Peter Kunszt (CERN)

SecretarySecretary: Alex Sim (LBNL): Alex Sim (LBNL)

http://sdm.lbl.gov/gsmhttp://sdm.lbl.gov/gsm

2 GGF- Grid Storage Management WG

IP and CopyrightIP and Copyright

Intellectual Property StatementThe GGF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the GGF Secretariat.The GGF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this recommendation. Please address the information to the GGF Executive Director.

Full Copyright NoticeCopyright (C) Global Grid Forum (2003). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the GGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the GGF Document process must be followed, or as required to translate it into languages other than English.The limited permissions granted above are perpetual and will not be revoked by the GGF or its successors or assigns.This document and the information contained herein is provided on an "AS IS" basis and THE GLOBAL GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."

3 GGF- Grid Storage Management WG

Purpose of WG: Develop Standards for Purpose of WG: Develop Standards for Grid Storage ManagementGrid Storage Management

Define the functional specification of:Define the functional specification of:

Storage Resource Managers (SRMs)Storage Resource Managers (SRMs)

DefinitionDefinition

SRMs are middleware components SRMs are middleware components

whose function is to provide dynamicwhose function is to provide dynamic

space allocationspace allocation file managementfile management

of shared storage components on the Gridof shared storage components on the Grid

4 GGF- Grid Storage Management WG

Basic IssuesBasic Issues

• Suppose you want to run a job on your local machineSuppose you want to run a job on your local machine• Need to allocate space• Need to bring all input files• Need to ensure correctness of files transferred• Need to monitor and recover from errors• What if files don’t fit space? Need to manage file streaming• Need to remove files to make space for more files

• Now, suppose that the machine and storage space is a Now, suppose that the machine and storage space is a shared resourceshared resource• Need to to the above for many users• Need to enforce quotas• Need to ensure fairness of space allocation and scheduling

5 GGF- Grid Storage Management WG

Basic IssuesBasic Issues

• Now, suppose you want to do that on a GridNow, suppose you want to do that on a Grid• Need to access a variety of storage systems• mostly remote systems, need at have access permission• Need to have special software to access mass storage systems

• Now, suppose you want to run distributed jobs on the Now, suppose you want to run distributed jobs on the GridGrid• Need to allocate remote spaces• Need to move (stream) files to remote sites• Need to manage file outputs and their movement to destination

site(s)

6 GGF- Grid Storage Management WG

Peer-to-Peer Uniform InterfacePeer-to-Peer Uniform Interface

MSS

Storage Resource Manager

network

clientClient

(command line)... Client’s site

...Disk

CacheDisk

Cache

Site 2Site 1 Site N

Storage Resource Manager

DiskCache

Storage Resource Manager

ClientProgram

DiskCache

DiskCache

...

Storage Resource Manager

DiskCache

DiskCache

...

Uniform SRMinterface

7 GGF- Grid Storage Management WG

General Analysis ScenarioGeneral Analysis Scenario

MSS

RequestExecuter

Storage Resource Manager

Metadatacatalog

Replicacatalog

NetworkWeatherService

logicalquery

network

clientclient ...

RequestInterpreter

requestplanning

A set oflogical files

Execution plan and site-specific

files

Client’s site

...Disk

Cache

DiskCache

ComputeEngine

DiskCache

Compute Resource Manager

Storage Resource Manager

ComputeEngine

DiskCache

Requests fordata placement andremote computation

Site 2Site 1 Site N

Storage Resource Manager

Storage Resource Manager

Compute Resource Manager

result files

ExecutionDAG

Uniform SRMinterface

8 GGF- Grid Storage Management WG

Motivation & RequirementsMotivation & Requirements

• Grid architecture needs to include reservation & Grid architecture needs to include reservation & scheduling of:scheduling of:• Compute resources• Storage resources• Network resources

• Storage Resource Managers (SRMs) role in the Storage Resource Managers (SRMs) role in the data grid architecturedata grid architecture• Shared storage resource allocation & scheduling• Specially important for data intensive applications• Often files are archived on a mass storage system (MSS)• Wide area networks – need to minimize transfers by file

sharing • Scaling: large collaborations (100’s of nodes,

1000’s of clients) – opportunities for file sharing• File replication and caching may be used• Need to support non-blocking (asynchronous) requests

9 GGF- Grid Storage Management WG

Part IPart I

ConceptsConcepts

10 GGF- Grid Storage Management WG

Storage Resource ManagementStorage Resource Management

• Main conceptsMain concepts

• Allocate spaces, space lifetime

• Get/put files from/into spaces

• Pin files for a lifetime

• Release files and spaces

• Get files into spaces from remote sites

• Manage directory structures in spaces

• SRMs communicate as peer-to-peer

• Negotiate transfer protocols

• No logical name space management

11 GGF- Grid Storage Management WG

HistoryHistory

• 4 year of Storage Resource (SRM) Management activity4 year of Storage Resource (SRM) Management activity

• Experience with system implementations v.1.x - 2001Experience with system implementations v.1.x - 2001• MSS: HPSS (LBNL, ORNL, BNL), Enstore (Fermi), JasMINE (Jlab),

Castor (CERN), MSS (NCAR), SE (RAL) …• Disk systems: DRM(LBNL), (dCache(Fermi), jSRM (Jlab), …

• SRM v2.x spec was finalized - 2003SRM v2.x spec was finalized - 2003

• Several implementations of v2.x completed or in-Several implementations of v2.x completed or in-progressprogress• Jlab, Fermi, CERN, LBNL

• Started GSM: GGF-BOF at GGF8 (June 2003)Started GSM: GGF-BOF at GGF8 (June 2003)

• Last SRM collaboration meeting – Sept. 2004Last SRM collaboration meeting – Sept. 2004

• SRM v3.0 spec (for GGF) being finalized - 2005SRM v3.0 spec (for GGF) being finalized - 2005

12 GGF- Grid Storage Management WG

CERNCERN: Peter Kunszt, Jean-Philippe Baud, Olof Barring, James: Peter Kunszt, Jean-Philippe Baud, Olof Barring, James Casey, Sebastien Ponce Casey, Sebastien Ponce

Rutherford labRutherford lab: Jens Jensen, Owen Synge, Shaun De Witt: Jens Jensen, Owen Synge, Shaun De Witt

Jefferson LabJefferson Lab: Bryan Hess, Andy Kowalski, Chip Watson, : Bryan Hess, Andy Kowalski, Chip Watson, Michael Haddox-SchatzMichael Haddox-Schatz

FermilabFermilab: Don Petravick, Timur Perelmutov: Don Petravick, Timur Perelmutov

LBNLLBNL: Junmin Gu , Arie Shoshani, Alex Sim, Kurt Stockinger: Junmin Gu , Arie Shoshani, Alex Sim, Kurt Stockinger

UnivaUniva: Rich Wellner: Rich Wellner

Current Storage Resource ManagementCurrent Storage Resource ManagementActive Working GroupActive Working Group

13 GGF- Grid Storage Management WG

Uniformity of Interface Uniformity of Interface Compatibility of SRMsCompatibility of SRMs

SRM SRM SRM

Enstore JASMine

ClientUSER/APPLICATIONS

Grid Middleware

SRM

dCache

SRM

Castor

SRM

Unix-baseddisks

SRM

SE

CCLRC RAL

14 GGF- Grid Storage Management WG

Where do SRMs belongWhere do SRMs belongin the Grid architecture?in the Grid architecture?

ComputeSystems

Networks

OtherStorage

systems

StorageResourceManager

ComputeResource

Management

General DataDiscoveryServices

CommunityAuthorization

Services

Application-Specific Data

Discovery Services

StorageManagement(Brokering)

ComputeScheduling(Brokering)

Data Filtering orTransformation

Services

DatabaseManagement

Services

RequestInterpretationand Planning

Services

File TransferService(GridFTP)

DataTransportServices

Monitoring/AuditingServices

Workflow orRequest

ManagementServices

Consistency Services(e.g., Update Subscription,Versioning, Master Copies)

DataFederationServices

RE

SO

UR

CE

:

CO

LLE

CT

I VE

1:

GE

NE

RA

LS

ER

VIC

ES

FO

RC

OO

RD

INA

TIN

GM

ULT

I PLE

RE

SO

UR

CE

S

CO

LLE

CT

IVE

2:

SE

RV

I CE

SS

PE

CIF

IC T

OA

PP

LIC

AT

ION

DO

MA

IN O

RV

IRT

UA

L O

RG

.

ResourceMonitoring/

Auditing

FA

BR

ICC

ON

NE

CT

I VIT

Y

CommunicationProtocols (e.g.,TCP/IP stack)

Authentication andAuthorization

Protocols (e.g., GSI)

Data Filtering orTransformation

Services

CO

LLE

CT

I VE

This figure based on theGrid Architecture paper by Globus Team

Mass StorageSystem(HPSS)

15 GGF- Grid Storage Management WG

SRMs supports data movement betweenSRMs supports data movement betweenstorage systemsstorage systems

ComputeSystems

Networks

OtherStorage

systems

StorageResourceManager

ComputeResource

Management

CommunityAuthorization

Services

Application-Specific Data

Discovery Services

DataCompute

Scheduling(Brokering)

Data Filtering orTransformation

Services

DatabaseManagement

Services

RequestInterpretationand Planning

Services

File TransferService(GridFTP)

DataTransportServices

Monitoring/AuditingServices

Workflow orRequest

ManagementServices

Consistency Services(e.g., Update Subscription,Versioning, Master Copies)

DataFederationServices

RE

SO

UR

CE

:S

HA

RIN

G S

ING

LER

ES

OU

RC

ES

CO

LLE

CT

I VE

1:

GE

NE

RA

LS

ER

VIC

ES

FO

RC

OO

RD

INA

TIN

GM

ULT

I PLE

RE

SO

UR

CE

S

CO

LLE

CT

IVE

2:

SE

RV

ICE

SS

PE

CIF

I C T

OA

PP

LIC

AT

ION

DO

MA

IN O

RV

IRT

UA

L O

RG

.

ResourceMonitoring/

Auditing

FA

BR

ICC

ON

NE

CT

I VIT

Y

CommunicationProtocols (e.g.,TCP/IP stack)

Authentication andAuthorization

Protocols (e.g., GSI)

CO

LLE

CT

I VE

This figure based on theGrid Architecture paper by Globus Team

Mass StorageSystem(HPSS)

General DataDiscoveryServices

Data Filtering orTransformation

ServicesMovement

Storage

16 GGF- Grid Storage Management WG

Concepts: Types of FilesConcepts: Types of Files

• Volatile: temporary files with a lifetime guaranteeVolatile: temporary files with a lifetime guarantee• Files are “pinned” and “released”• Files can be removed by SRM when released or when

lifetime expires

• PermanentPermanent• No lifetime• Files can only be removed by creator (owner)

• Durable: files with a lifetime that CANNOT be Durable: files with a lifetime that CANNOT be removed by SRMremoved by SRM• Files are “pinned” and “released”• Files can only be removed by creator (owner)• If lifetime expires – invoke administrative action (e.g. notify

owner, archive and release space)

17 GGF- Grid Storage Management WG

Concepts: Types of StorageConcepts: Types of Storageand spacesand spaces

• TypesTypes• On-line

• Storage where files are moved to before their use • Near-line

• Requires latency before files can be accessed• Remote files (non-local to the SRM) are considered near-line

• Retention qualityRetention quality• Custodial (High quality) • Output (Middle quality) • Replica (low Quality)

• Spaces can be reserved in these storage typesSpaces can be reserved in these storage types• Spaces can be reserved for a lifetime• No limit on number of spaces• Space reference handle is returned to client• Total space of each type are subject to SRM and/or VO policies

• Assignment of files to spacesAssignment of files to spaces• Files can be assigned to any space, provided that their lifetime

expiration is shorter than the lifetime expiration of the space

18 GGF- Grid Storage Management WG

Concepts: managing spacesConcepts: managing spaces

• Default spacesDefault spaces• Files can be put into an SRM without explicit reservation• Defaults are not visible to client

• Files are placed in storage according to their typeFiles are placed in storage according to their type• Volatile files usually placed in on-line storage• Permanent files usually placed in near-line storage• Durable files usually placed in on-line storage, but if

lifetime expires are moved to near-line storage• Compacting spaceCompacting space

• Release all unused space – space that has no files or files whose lifetime expired

• Releasing spaceReleasing space• Release all files whose lifetime has not expired• Compact space• Move permanent and durable files to near-line storage• Release remaining space

19 GGF- Grid Storage Management WG

Concepts: Directory ManagementConcepts: Directory Management

• Usual unix semanticsUsual unix semantics• srmLs, srmMkdir, srmMv, srmRm, srmRmdir, srmCp

• A single directory for all spacesA single directory for all spaces• No directories for each file type• File assignment to spaces is virtual

• Access control servicesAccess control services• Support owner/group/world permission

• Can only be assigned by owner• When file requested by user, SRM should check permission

with source site

20 GGF- Grid Storage Management WG

Examples of Directory StructuresExamples of Directory Structures(user defined)(user defined)

D1

D3D2

D4

F2 (P)

F4 (P) F5 (D)

F1 (D) F3 (V)

D1

D3D2D4

F1 (V) F2 (V) F3 (V) F4 (D) F5 (D) F6 (D) F7 (P) F8 (P)

(1) Mixed file types (2) By file type

• Supported function: ChangeFileType

• Advantage of (1): no need to move files when file types are changed

21 GGF- Grid Storage Management WG

Concepts: Space ReservationsConcepts: Space Reservations

• NegotiationNegotiation• Client asks for space: C-guaranteed, MaxDesired• SRM return: S-guaranteed <= C-guaranteed,

best effort <= MaxDesired

• Type of spaceType of space• Can be specified• Subject to limits per client (SRM or VO policies)• Default: volatile

• LifetimeLifetime• Negotiated: C-lifetime requested• SRM return: S-lifetime <= C-lifetime

• Reference handleReference handle• SRM returns space reference handle• User can provide: srmSpaceTokenDescription to recover handles

22 GGF- Grid Storage Management WG

Concepts: Transfer Protocol NegotiationConcepts: Transfer Protocol Negotiation

• NegotiationNegotiation• Client provides an ordered list• SRM return: highest possible protocol it supports

• ExampleExample• Protocols list: bbftp, gridftp, ftp• SRM returns: gridftp

• AdvantagesAdvantages• Easy to introduce new protocols• User controls which protocol to use• Default – SRM policy choice

• How it is returned?How it is returned?• The protocol of the Transfer URL (TURL)• Example: bbftp://dm.slac.edu/temp/run11/File678.txt

23 GGF- Grid Storage Management WG

Concepts: Site URL and Transfer URLConcepts: Site URL and Transfer URL

• Provide: Site URL (SURL)Provide: Site URL (SURL)• URL known externally – e.g. in Replica Catalogs• e.g. srm://sleepy.lbl.gov:4000/tmp/foo-123

• Get back: transfer URL (TURL)Get back: transfer URL (TURL)• Path can be different than SURL – SRM internal mapping• Protocol chosen by SRM based on request protocol preference• e.g. gridftp://dm.lbl.gov:4000/home /level1/foo-123

• One SURL can have many TURLOne SURL can have many TURL• Files can be replicated in multiple storage components• Files may be in near-line and/or on-line storage

• In light-weight SRM (a single file system on disk)In light-weight SRM (a single file system on disk)• SURL can be the same as TURL except protocol

• File sharing is possibleFile sharing is possible• Same physical file, but many requests• Needs to be managed by SRM

24 GGF- Grid Storage Management WG

Concepts: Multi-file requestsConcepts: Multi-file requests(using site URL and Transfer URL)(using site URL and Transfer URL)

• Can srmRequestToGet multiple filesCan srmRequestToGet multiple files• Required: array of SURLs• Optional: space file type, space handle, Protocol list• Optional: total retry time

• Managing request queueManaging request queue• Allocate space according to policy, system load, etc.• Bring in as many files as possible• Provide information on each file brought in or pinned• Bring additional files as soon as files are released• Support file streaming

Requested files in near-line storage

Cached files in on-line storage

File streaming:stage, use,release, stage next

25 GGF- Grid Storage Management WG

Part IIPart II

SRM functions:SRM functions:

Core and Advanced FeaturesCore and Advanced Features

26 GGF- Grid Storage Management WG

SRM MethodsSRM Methods

File MovementsrmPrepareToGetsrmPrepareToPutsrmRemoteCopy Lifetime managementsrmReleaseFilessrmPutDonesrmExtendFileLifeTime

Terminate/resumesrmAbortRequestsrmAbortFilesrmSuspendRequestsrmResumeRequest 

Space managementsrmReserveSpacesrmReleaseSpacesrmUpdateSpace FileType managementsrmChangeFileStorageType

Status/metadatasrmGetRequestStatussrmGetFileStatussrmGetRequestSummarysrmGetRequestIDsrmGetFilesMetaDatasrmGetSpaceMetaData

27 GGF- Grid Storage Management WG

SRM v3.0: Core vs. Advanced Features (1)SRM v3.0: Core vs. Advanced Features (1)

• File movementFile movement• PrepareToGet• PrepareToPut• RemoteCopy

• Request capabilitiesRequest capabilities• Multi-file Streaming• Trans. Prot. Negotiation• File lifetime negotiation

• File Storage typesFile Storage types• Volatile• Permanent• Durable

Core ADVANCEDCore ADVANCED

yes yesyes yesno yes

yes yesyes yes

no yes

yes yesyes yes

optional yes

28 GGF- Grid Storage Management WG

SRM v3.0: Core vs. Advanced Features (2)SRM v3.0: Core vs. Advanced Features (2)

Core ADVANCEDCore ADVANCED• Space reservationsSpace reservations

• Space-time negotiation• Storage types

• Remote accessRemote access• gridFTP• Other SRMs

• DirectoryDirectory• ls• mkdir, rmdir• cp, mv, rm

• Terminate/suspendTerminate/suspend• Abort file• Abort request• Suspend/resume request

no yesno yes

no yesno yes

noyeslimitedyes

no yes

yes yesyes yesno yes

29 GGF- Grid Storage Management WG

Advanced Functions are groupedAdvanced Functions are groupedinto Feature Setsinto Feature Sets

• Core SetCore Set

• Advanced Feature SetsAdvanced Feature Sets• Remote Access• Space Management• Directory Management• Authorization• Request Administration

30 GGF- Grid Storage Management WG

Extending functions for advance featuresExtending functions for advance features

• Issue: what to do in the case of functions that are in core, Issue: what to do in the case of functions that are in core, but are also used in advanced features with additional functionalitybut are also used in advanced features with additional functionality

• E.g. Specifying a space-token for srmRemoteCopy when “space reservation” feature is supported.

• Solution: Extending parameters according to features

f1 (C)f3 (C)

f7 (C)

f8(C)

f4 (C)f2 (C)

f5 (C)

f6 (C)

f1 (F3)

f2 (F3)

CORE

FEATURE 1

FEATURE 2

FEATURE 3

FEATURE 4

FEATURE 5

FEATURE 6

f1 (F3)

f2 (F3)f1 (F1)

f2 (F1)

f3 (F1)

f1 (F3)f2 (F3)

f3 (F3)

f4(F3)

31 GGF- Grid Storage Management WG

Core FunctionsCore Functions

srmPrepareToGetsrmReleaseFilessrmStatusOfGetRequest

srmPrepareToPutsrmPutFileDonesrmPutRequestDonesrmStatusOfPutRequest

srmLssrmLsStatus

srmRmsrmRmStatus

srmExtendRequestedFileLifetimesrmChangeFileStorageTypesrmChangeFileStorageTypeStatus

srmAbortRequestsrmAbortRequestedFiles

srmGetRequestSummarysrmGetRequestTokenssrmGetSRMStorageInfosrmGetTransferProtocols

srmGetFeatures

32 GGF- Grid Storage Management WG

Advanced Feature setsAdvanced Feature sets

Space ManagementsrmReserveSpacesrmUpdateSpacesrmReleaseSpace

srmCleanupFilesFromSpace

srmGetSpaceMetaDatasrmGetSpaceTokens

Remote AccesssrmRemoteCopysrmStatusOfRemoteCopy

Directory ManagementsrmCpsrmCpStatussrmMvsrmMvStatussrmMkdirsrmRmdir

AuthorizationsrmCheckPermissionsrmSetPermission

Request AdministrationsrmResumeRequestsrmSuspendRequest

33 GGF- Grid Storage Management WG

e.g. Space Reservation Functional Spece.g. Space Reservation Functional Spec

srmReserveSpacesrmReserveSpaceIn: In: TUserIDTUserID userID,userID,

TSpaceTypeTSpaceType typeOfSpace, typeOfSpace, StringString userSpaceTokenDescription,userSpaceTokenDescription,TSizeInBytesTSizeInBytes sizeOfTotalSpaceDesiredsizeOfTotalSpaceDesired,, TSizeInBytesTSizeInBytes sizeOfGuaranteedSpaceDesired,sizeOfGuaranteedSpaceDesired,TLifeTimeInSecondsTLifeTimeInSeconds lifetimeOfSpaceToReserve,lifetimeOfSpaceToReserve,TStorageSystemInfoTStorageSystemInfo storageSystemInfostorageSystemInfoInt Int expectedFileSize [ ]expectedFileSize [ ]

Out: TSpaceToken, Out: TSpaceToken, referenceHandleOfReservedSpacereferenceHandleOfReservedSpace,,TSpaceTypeTSpaceType typeOfReservedSpace, typeOfReservedSpace, TSizeInBytesTSizeInBytes sizeOfTotalReservedSpace,sizeOfTotalReservedSpace,TSizeInBytesTSizeInBytes sizeOfGuaranteedReservedSpace,sizeOfGuaranteedReservedSpace,TLifeTimeInSecondsTLifeTimeInSeconds lifetimeOfReservedSpace, lifetimeOfReservedSpace, TReturnStatusTReturnStatus returnStatusreturnStatus

34 GGF- Grid Storage Management WG

e.g. Prepare-to-Get Functional Spece.g. Prepare-to-Get Functional Spec

srmPrepareToGetsrmPrepareToGet

In:In: TUserIDTUserID userID,userID,

TGetFileRequest[ ]TGetFileRequest[ ] arrayOfFileRequest,arrayOfFileRequest,

string[]string[] TransferProtocols,TransferProtocols,

stringstring userRequestDescription,userRequestDescription,

TStorageSystemInfoTStorageSystemInfo storageSystemInfo, storageSystemInfo,

BooleanBoolean streamingModestreamingMode

Out:Out: TRequestTokenTRequestToken requestToken,requestToken,

TReturnStatusTReturnStatus returnStatus,returnStatus,

TGetRequestFileStatus[ ]TGetRequestFileStatus[ ] arrayOfFileStatusarrayOfFileStatus

35 GGF- Grid Storage Management WG

e.g. “TGetFileRequest” typedef e.g. “TGetFileRequest” typedef

Typedef struct {stringTypedef struct {string fromSURL,fromSURL,

intint lifetime, // pin timelifetime, // pin time

TFileStorageTypeTFileStorageType fileStorageType,fileStorageType,

stringstring spaceToken,spaceToken,

intint knownFileSize,knownFileSize,

booleanboolean recursive,recursive,

intint numberOfRecursiveLevels numberOfRecursiveLevels

} TGetFileRequest} TGetFileRequest

36 GGF- Grid Storage Management WG

e.g. srmGetFeaturese.g. srmGetFeatures

• Discovery function for clients to find out which features Discovery function for clients to find out which features an SRM supportsan SRM supports• EnumFileStorageType fileStorageType[],

EnumSpaceType spaceType[],EnumSRMFeatures supportedFeature[]

• SRMFeatures := SRM_CORE SRM_REMOTE_ACCESS

SRM_SPACE_MANAGEMENT

SRM_DIRECTORY_MANAGEMENT SRM_AUTHORIZATION

SRM_ADMINISTRATION SRM_ACCOUNTING

37 GGF- Grid Storage Management WG

e.g. srmGetSRMStorageInfoe.g. srmGetSRMStorageInfo

• To retrieve SRM storage info, such as storage capacity, user quota, To retrieve SRM storage info, such as storage capacity, user quota, etc.etc.• Input

• EnumStorageAttributes desiredAttributes[] • Output

• SupportedAttributes {EnumStorageAttributes storageAttr,String value,String valueType

} []• EnumStorageAttributes :=  

SRM_STORAGE_TYPE SRM_STORAGE_CAPACITY SRM_STORAGE_ACCESS_TYPE SRM_USER_STORAGE_MAX SRM_USER_STORAGE_MINSRM_USER_STORAGE_DEFAULT_LIFETIME SRM_DEFAULT_FILE_LIFETIME SRM_DEFAULT_FILE_STORAGE_TYPE SRM_REQUEST_INFO_AVAILABILITY ….

38 GGF- Grid Storage Management WG

SummarySummary

• Storage Resource Management – essential for GridStorage Resource Management – essential for Grid

• SRM is a functional definitionSRM is a functional definition• Adaptable to different frameworks (WS, WSRF, …)

• Multiple implementations interoperateMultiple implementations interoperate• Permit special purpose implementations for unique products• Permits interchanging one SRM product by another

• SRM implementations exist and some in production useSRM implementations exist and some in production use• Particle Physics Data Grid• Earth System Grid• More coming …

• Cumulative experience in GGF-WGCumulative experience in GGF-WG• Specifications SRM v3.0 complete