srm v2.1: additional design issues

Arie Shoshani – Dec 2002

Participants:Participants:

meeting held at CERNmeeting held at CERNDecember, 2002December, 2002

JLABJLAB: Bryan Hess, Andy Kowalski: Bryan Hess, Andy Kowalski

FermiFermi: Don Petravick, Timur Perelmutov: Don Petravick, Timur Perelmutov

LBNLLBNL: Alex Sim, Arie Shoshani: Alex Sim, Arie Shoshani

WP2WP2: Peter Kunszt, Heinz Stockinger,: Peter Kunszt, Heinz Stockinger,

Kurt Stockinger, Erwin LaureKurt Stockinger, Erwin Laure

WP5WP5: Jean-Philippe Baud, Stefano Occhetti, : Jean-Philippe Baud, Stefano Occhetti, Jens Jensen, Emil Knezo, owen synge Jens Jensen, Emil Knezo, owen synge

SRM V2.1:SRM V2.1:Additional Design IssuesAdditional Design Issues


Agenda

• Goal— Agree on open SRM issues— Main issues

• Space reservations• Directory functions• Access control• Lifetime negotiations (space and file)

— Other issues• Meaning of lifetime• Renew lifetime• Revisit srmCopy (push mode)• Revisit call-backs with WSDL handles

• Architecture— Where do SRM fit?


High Level View of Demo Setup

SRM

SRM

SRM

Enstore

JASMine

Client(USER/APPLICATIONS)

CASTOR SRM V2.0

SRM V1.1


How Was It Done?only MSS-specific module modified

Disk ResourceManager (DRM)

Tape ResourceManager (TRM)

HPSS-specificAccess Module

DiskCache

HRM-HPSS

Disk ResourceManager (DRM)

Tape ResourceManager (TRM)

NCAR-specificAccess Module

DiskCache

HRM-HPSS

Specialize for NCAR-MSS


SRMs in Action : PPDGSRMs in Action : PPDG

DiskCache

DiskCache

HRM-COPY(thousands of files)

HRM-GET (one file at a time)

HRM-ClientCommand-line Interface

HRM(performs writes)

HRM(performs reads)

LBNL BNL

GridFTP GET (pull mode)

Anywhere

stage filesarchive files

Network transfer


Web-Based File Monitoring ToolWeb-Based File Monitoring Tool

Shows:-Files already transferred- Files during transfer- Files to be transferred

Also shows foreach file:-Source URL-Target URL-Transfer rate


SRMs in Action : PPDGSRMs in Action : PPDG

DiskCache

DiskCache

HRM-COPY(thousands of files)

HRM-GET (one file at a time)

HRM-ClientCommand-line Interface

HRM(performs writes)

HRM(performs reads)

LBNL BNL

GridFTP GET (pull mode)

Anywhere

stage filesarchive files

Network transfer


Web-Based File Monitoring ToolWeb-Based File Monitoring Tool

Shows:-Files already transferred- Files during transfer- Files to be transferred

Also shows foreach file:-Source URL-Target URL-Transfer rate


Recent Measurements Recent Measurements of large multi-file replicationof large multi-file replication

Shows that the network is the bottleneck


HRMs and GridFTP

HRM

GridFTP

SRM-API

GridFTP-API

Client

HRM

GridFTP move

SRM-API

GridFTP-API

Client

Using HRM protocol New: GridFTP-HPSSthrough HRM

GridFTP entry


GridFTP-HRM-Layerimplementation detail

HRM

GridFTP-API

Client

GridFTP entry

GridFTP move

GridFTP exit

FTP-HRMLayer

Sharedmemory Corba

1a 1b

2a 2b

3a 3b

1a: stor/retv1b: hrm_get/hrm_put

2b: call_back2a: unblock semaphore

3a: success_code3b: hrm_release


Current version proposalCurrent version proposal

byby

Alex Sim, Junmin Gu, Arie ShoshaniAlex Sim, Junmin Gu, Arie Shoshani

SRM V2.1:SRM V2.1:Additional Design IssuesAdditional Design Issues


Request Manager and SRMs

tape system

HRM

RequestExecuter

DRM

DiskCache

property-file index

Replicacatalog

NetworkWeatherService

logicalquery

pinning & filetransfer requests

network

DRM

DiskCache

clientclient ...

RequestInterpreter

requestplanning

logical files

site-specific files

Client’s site

...

DiskCache

site-specific files requests


How to do direct file access

• Now: — use srmGet with desired protocol, e.g. rfio, dcap

(desy), file:, globus-xio— Expect to call srmRelease when done— Explore: how to use this with “redirect”

• Agreed to have at least one common protocol:— Gridftp for file transfer— ??? For direct access (should be POSIX compliant)


Space Reservation

• Space types— Option 1: volatile, durable, permanent— Option 2: volatile, permanent

• Why support “durable” space?— Reminder: durable files are temporary files that can

only be removed by owner/administrator— For systems that do not intend to have permanent

space (e.g. DRMs), but wish to support durable files

• Recommendation— Support space reservations for all three types— Size of reservation is negotiable— Request for additional space possible— Lifetime of reservation is negotiable— Request for lifetime extension possible


File assignment to spaces

• Implications— Need to acquire durable/permanent space to store durable files

— Need to acquire permanent space to store permanent files

— Can change file types within a space according to above figure

Volatile durable permanent

Volatile durable permanent SpaceTypes

FileTypes


Semantics of spaces

• File assignment— File lifetime cannot exceed space lifetime

• Reservation guarantees— Volatile: soft guarantee – space can be reclaimed, minimum is

guaranteed— Durable, permanent – hard guarantee

• Space removal/release— Permanent – by client

• ReleaseSpace – requires all files in space released• ForceReleaseSpace – remove all files as well

— Release pins, reclaim space if needed

— Durable – by client or administrator if lifetime expires• ReleaseSpace – requires all files in space released (files with lifetime

expired are not considered “released”)• ForceReleaseSpace – remove all files as well

— Release pins, reclaim space if needed

— Volatile• Assign minimum quota if request is still active• Otherwise, reclaim space


Guaranteeing quotas

• Add the concept of Durable – “best effort”— In addition to guaranteed

• Volatile is always “best effort”• Managing quotas – is an SRM choice

— At first, start with BE, ignore quota— SRM adjust quotas and other policies— VO responsibility SRM should have reporting

capabilities


Parameters of space reservation request / negotiations

Function: srmReserveSpace• In parameters

— User_ID— Type of space— Size (granularity: MBs)— Lifetime (granularity: minutes)

• Return parameters— Space_size_approved (MBs)— Lifetime_approved (minutes)— Request_ID— Hold lifetime (minutes)

Function: srmConfirmSpace• In parameters

— User ID— Request ID

• Return parametrs— OK, refuse


Space request functions

• srmReserveSpace()• srmReleaseSpace() option:force• srmResizeSpace()

— Total space desired

• srmCompactSpace() option:dynamic— Reduce space to size of active files (non-released files)— Reduce space as soon as files get released— If srmResizeSpace

• srmGetCurrentSpace()

• Comments:— Do not use /durable etc. as a way to address the space.

• E.g. srmPut (UserID=john, dir=xyz file=foo, space=durable)

— UserID not needed if security system is used (optional)— StorageAuthenInfo can be optionally provided (optional)


Space Usage Reporting

Reporting granularity

per period?

Per event?

per file? Per request?• Space used (in MB-Minutes)• Space reserved (in MB-Minutes)• Usage level (in MBs) - optional

— Total MBs of data transferred on behalf of user associated with space

• MBs are binaries – 1024x1024 bytes


File Sharing in Spaces

• Files (owned) in any spaces can be shared by requesters— Provided that requester has permission to “read”

• In volatile space— File size is counted against requester’s quota

• In durable / permanent space— Requester does not have to have durable / permanent space

— File size not counted against requester’s quota

— File is not moved to requester’s space

— If owner removes / releases file, wait till second client releases file or lifetime expired

• And no new clients can get access to it

• srmReassignToUser()— Owner give permission to reassign directory/file to another user

— Lifetime should be specified

— Space continues to be charged to owner till srmMoveTo Space happens

• srmMoveToSpace()— move from: directory of in one space to directory into requerter’s space

— Can designate a file type different from original

— Recommendation for implementation: do not copy files


Directory Semantics

• Client can establish directories for any of their spaces— Including “volatile” spaces !— File sharing in “volatile spaces” must be supported

(implementation choice – e.g. links)

• Only one space type per client !• Clients refer to spaces as:

— volatile, durable, permanent (reserved terms)— Top level directory names is local DRM choice

• E.g. /home/srm/john/durable

• TFNs (in TURLs) returned to clients include 4 levels + user established directories

• E.g. /home/srm/john/durable/abc/xyz/file.foo(here abc, xyz are directories john setup)


Directory Operations

• ls, mkdir, mv, cd, rm, rmdir, cd, pwd— (Prefix with srm?)— Relative to space name: /volatile, /durable, /permanent— Default is /volatile— Allow –r (recursive) in rm, rmdir, ls— Allow for ls: max_list, offset (Chip Watson)

• Rm— Hard delete in /durable or /permanent space— Soft delete (advisory delete) in /volatile

• Access control— In future: all access through CAS— In current version:

• Defaults— Volatile: read-world— Durable / permanent: read-owner

• User can use chmod — Volatile – chmod may be refused by SRM


Lifetime Issues

• Option 1: Relative time since response to request is made?— e.g. 30 min from now— what if communication is slow?— Agreed to have relative time

• Option 2: Absolute time— how to synchronize clocks— Agreed to have creation time with GMT (best if synchronized with

NTP or atomic clock)

• Lifetime granularity – Keep it seconds to simplify implementation• Should a lifetime of a file be negotiable? Yes

— Per request? Yes-optional (default applies)— Per file? Yes-optional (default applies)— Per space? Yes-optional (default applies)— If so, need to add confirmation – no confirmation, client can release

• Alternative: not negotiable (covered by above)— Volatile files: srm’s choice— Durable: client’s choice (not exceeding space lifetime)


Security Issues

• How to support multiple security models —GSI, Kerberos, SSL, etc… over http

• Need a common ground (GSI)—GSI over HTTP for communication - OK—GridFTP (gsiftp) for data transfer – OK— currently you need to be a user of the system to put files in

that system—Can also be done by group ID, and SRM manage quotas—E.g. expereince with GSI–kerboros map (at Fermi?)

• How to deal with systems that have their own internal security : optional StorageSystemID—E.g. HPSS at NERSC

• Who is enforcing file access permissions in a Virtual Organization (VO)— If CAS in the future, what mechanism?—How do SRMs report usage to CAS?


Revisit Features: srmGet, srmPut, srmCopy

SRM Client

Client-FTP-get(pull)

Client-FTP-put(push)

srmGet/srmPut

SRM-FTP-put(push)

SRM ClientSRM/

No-SRMSRM-FTP-get

(pull)

srmCopy

SRM/No-SRM

FTP-get


Revisit Features: get, put

• srmGet, srmPut: relative to client— srmGet: (Client wants to pull a file from the SRM)

• If file already in SRM space => pin, return location• If file not in SRM space => allocate space, get file

from its source location, pin, call-back when file arrives

• New: Can specify location in a directory, file-type, space-type

• Followed by srmRelease— srmPut: (Client wants push a file into SRM)

• Allocate space, return location• New: Can specify location in a directory, file-type,

space-type• Followed by srmPutDone


Revisit Features: copy

• srmCopy: relative to SRM— srmCopyPull: Copy from remote location to SRM

• Call-back when done• Option: notify remote source to release file when

done— srmCopyPush: Copy from SRM to remote location

(add?)• Call-back when done• Option: Release file when transfer done

— Example scenario: ask an HRM to push file to a site that have only GridFTP

— Discussion: Don Petravick


Revisit Features: directory functionality

• Should we allow specification of a directory for srmGet, srmPut, srmCopy?— srmGet

• Now: srmGet {LFN, SURL}

• Add: srmGet (Dir=path/directory_name)

• Recursive? No

— srmPut

• Now: srmPut {LFN, StURL}

• Add: srmPut [{LFN}, StDir= path/directory_name]

• Recursive? No

— srmCopy

• Now: srmCopy {source-URL}

• Add: srmCopy (source-URL, target-StDir)

• Add: srmCopy (source-StDir, target-StDir)

• Recursive? - srmCopy (source-StDir, target-StDir, - r)


Revisit Features: LFNs

• LFN can be very long— Long path name, long filename

• Any ideas— SRMs could assign unique IDs internally— Should IDs be visible externally, like RDBMSs?


Revisit Features: others

• “Call_backs" through WSDL – handle— Since supported by WSDL, why not include in Basic Version? –

OK, but optional

• File and system status functionality— Add: which files in file-set are currently in SRM cache- Yes— what should be reported that is useful for planning?— what is useful for advertising to RepCat and Info-service?

• SRM should register itself with Info-service• SRM should be notifying RepCat on deletes• Flag for registering in RepCat

— should status be active or passive?

• Add max-file-size to get/put/copy requests?— Expect the system to allocate max-file-length and adjust space

after file is transferred— Expect system to kill transfer if file size exceeds max-file-

length - OK, optional

• Remove “srm” from all commands except get, put, copy? srmPrepareGet, srmPreparePut, srmCopy


New feature: include Replica Catalog connectivity

• Replica Catalogs— RLS, Jlab-RepCat, …— Private catalogs: STAR-catalog, Babar-SRB, …

• Replica management— Globus: grid-ftp + RLS— Desirable: SRM + some catalog

• File Replication Service— Robust copying of large number of files (1000’s)— Automatic recovery from transient failures— Logs, dynamic tracking and monitoring— Automatic registration to replica catalogs

• Add: post-hook?— After each file gets transferred, call post-hook service

• (Also add: pre-hook?)— Call filtering function before file is transferred


SOAP Inter-Operational issue

• SOAP implementations are not compatible in complex data types, specially with string array and struct array.— Most array test failures are due to the lack of support for

SOAP sparse array and partially transmitted array encodings. Array tests with 'regular' arrays are mostly okay and interoperable

• Need a common ground: OGSA/GT3— In the meantime, use SOAP over https-with-GSI

— Implies: GSI using minimum security model


What else?

• Issues we do not want to discuss now, but may be important in the future— Interacting with RLS and other catalogs— Pre/post processing hooks— Advertising SRM capabilities— Providing soft guarantees for planning— …


SOAP InterOp test

StringArray

IntegerArray

FloatArray

StructArray

2D StringArray

NestedStruct

NestedArray

StringArray

IntegerArray

FloatArray

StructArray

StringArray

IntegerArray

FloatArray

StructArray

ApacheSoap 2.2 FAULT PASS FAULT FAULT FAULT FAULT FAULT PASS PASS PASS PASS PASS PASS PASS PASS

Apache Axis PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

ASP .NET FAIL PASS PASS FAIL FAULT PASS FAIL PASS PASS PASS PASS PASS PASS PASS PASS

Easy Soap++ PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

GLUE FAIL PASS PASS FAIL FAULT PASS FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL

gSOAP PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

IONA XMLBus PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

MSSOAP3.0 FAULT PASS FAULT FAULT FAULT PASS FAULT PASS PASS PASS PASS PASS PASS PASS PASS

nuSOAP FAIL PASS PASS FAIL FAULT FAULT FAIL FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT

PEARSOAP FAIL PASS PASS FAIL FAULT PASS FAIL PASS PASS PASS PASS PASS PASS PASS PASS

SOAP4R PASS PASS PASS PASS PASS PASS PASS

SOAP::Lite PASS PASS PASS PASS FAULT PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

Spheon jSOAP PASS PASS PASS PASS FAULT PASS PASS FAULT FAULT FAULT FAULT FAULT FAULT FAULT FAULT

SQLData PASS PASS FAULT FAULT FAIL PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

White Mesa FAULT PASS FAULT PASS FAULT PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS

xSOAP FAULT PASS FAULT FAULT

Sun JAX RPC PASS PASS PASS PASS PASS PASS PASS PASS

SOURCE http://www.cs.fsu.edu/~engelen/soap.html with gSOAP clienthttp://www.apache.org/~rubys/ApacheClientInterop.html With

Apache SOAP client

http://www.apache.org/~rubys/ApacheClientInterop.htmlWith

Apache Axis clients

Test according to http://www.whitemesa.com/interop/proposal2.html and http://www.whitemesa.com/interop/proposalB.html


1) Where do SRMs belong1) Where do SRMs belongin the Grid architecture?in the Grid architecture?

ComputeSystems

Networks

OtherStorage

systems

StorageResourceManager

ComputeResource

Management

General DataDiscoveryServices

CommunityAuthorization

Services

Application-Specific Data

Discovery Services

StorageManagement(Brokering)

ComputeScheduling(Brokering)

Data Filtering orTransformation

Services

DatabaseManagement

Services

RequestInterpretationand Planning

Services

File TransferService(GridFTP)

DataTransportServices

Monitoring/AuditingServices

Workflow orRequest

ManagementServices

Consistency Services(e.g., Update Subscription,Versioning, Master Copies)

DataFederationServices

RE

SO

UR

CE

:S

HA

RIN

G S

ING

LER

ES

OU

RC

ES

CO

LLE

CT

I VE

1:

GE

NE

RA

LS

ER

VIC

ES

FO

RC

OO

RD

INA

TIN

GM

ULT

I PLE

RE

SO

UR

CE

S

CO

LLE

CT

IVE

2:

SE

RV

ICE

SS

PE

CIF

IC T

OA

PP

LIC

AT

ION

DO

MA

IN O

RV

IRTU

AL

OR

G.

ResourceMonitoring/

Auditing

FA

BR

ICC

ON

NE

CTI

VIT

Y

CommunicationProtocols (e.g.,TCP/IP stack)

Authentication andAuthorization

Protocols (e.g., GSI)


Services

CO

LL

EC

TI V

E

This figure based on theGrid Architecture paper

(new version)

Mass StorageSystem(HPSS)


ComputeSystems

Networks

OtherStorage

systems


ComputeResource

Management



Services


Discovery Services




Services

DatabaseManagement

Services


Services




Workflow orRequest

ManagementServices



RE

SO

UR

CE

:S

HA

RIN

G S

ING

LER

ES

OU

RC

ES

CO

LLE

CT

I VE

1:

GE

NE

RA

LS

ER

VIC

ES

FO

RC

OO

RD

INA

TIN

GM

ULT

I PLE

RE

SO

UR

CE

S

CO

LLE

CT

IVE

2:

SE

RV

ICE

SS

PE

CIF

IC T

OA

PP

LIC

AT

ION

DO

MA

IN O

RV

IRTU

AL

OR

G.

ResourceMonitoring/

Auditing

FA

BR

ICC

ON

NE

CTI

VIT

Y





Services

CO

LL

EC

TI V

E



(new version)




ComputeSystems

Networks

OtherStorage

systems


ComputeResource

Management



Services


Discovery Services




Services

DatabaseManagement

Services


Services




Workflow orRequest

ManagementServices



RE

SO

UR

CE

:S

HA

RIN

G S

ING

LER

ES

OU

RC

ES

CO

LLE

CT

I VE

1:

GE

NE

RA

LS

ER

VIC

ES

FO

RC

OO

RD

INA

TIN

GM

ULT

I PLE

RE

SO

UR

CE

S

CO

LLE

CT

IVE

2:

SE

RV

ICE

SS

PE

CIF

IC T

OA

PP

LIC

AT

ION

DO

MA

IN O

RV

IRTU

AL

OR

G.

ResourceMonitoring/

Auditing

FA

BR

ICC

ON

NE

CTI

VIT

Y





Services

CO

LL

EC

TI V

E


(new version)



• GGF – standards• GLUE• Common minimal grounds

• Security• Misc.• GLUE


Basic functionality

• Space types— Volatile at minimum— Permanent, durable is optional— Durable: best effort, guaranteed

• Space reservations— Space reservation allowed

• Directory support— Yes

• srmMoveDir— yes

• Security— Gsi over http

• Transfer protocols— gridFTP

• srmChangeFileType— Provided that space of the requested type was acquired

• srmCopy— Both push and pull


Action Items (1)

• Canonical specification by Arie, everyone check• WSDL v.2.1 : generated by Timur, Alex, Bryan, …• Java translation by Timur• C/C++ translation by Alex• Simple Java client/server based on WSDL v.2.1 compatible

with httpg/OGSA/GT3 by Timur• Simple C++ client/server based on WSDL v.2.1• compatible with httpg/OGSA/GT3 by Alex• LBNL (Alex) will provide/maintain a repository for

documentations, WSDL and sample packages• Also LBNL will run test servers for both Java and C/C++

—http://sdm.lbl.gov/srm-wg will contain information

• Interface with SRM from the EDG Replica Manager (Reptor)- Heinz and Kurt


Action Items (2)

• SRM mailing list to be maintained : by Timur• Participating in GGF as a BoF/RG/WG – Peter to

clarify with Peter Clarke• Annual/bi-annual SRM workshop

— Next meeting ?

srm v2.1: additional design issues

Documents