A. Sim, CRD, L B N L 1Oct. 23, 2008

BeStManExtra Slides

A. Sim, CRD, L B N L 2Oct. 23, 2008

BeStMan Features

• Multiple disk partition support• Default space management for files with lifetime

• Allocation of space, garbage collection• Dynamic space reservation• Support for multiple file transfer servers

• E.g. configure for 5 GridFTP servers• Per-user (not per-request) request management• Multi-file/directory per request• Incoming and outgoing file transfer queue management and transfer

monitoring• Support file sharing and file streaming• Easy adaptability to Unix-based file systems• Easy adaptability and integration to special project environments• API support for customization for local mass storage systems• Simple installation and easy maintenance

A. Sim, CRD, L B N L 3Oct. 23, 2008

Berkeley Storage Manager (BeStMan)

• Java implementation• Designed to work with Unix-based

disk file systems • As well as MSS to stage/archive

from/to its own disk• Adaptable to other special file

systems and storages (e.g. NCAR MSS, SLAC XrootdFS)

• Uses in-memory database (BerkeleyDB)

Request Processing

MSS Access Management(PFTP, HSI, SCP...)DISK Management

Network Access Management(GridFTP. FTP, BBFTP, SCP... )

Request Queue Management Security Module

Local

Policy

Module

• Multiple transfer protocol support• Space reservation• Directory management• Can copy files from/to remote SRMs/GridFTP servers• Can copy entire directory recursively

• Large scale data movement of thousands of files• Recovers from transient failures (e.g. MSS maintenance, network down)

• Local Policy• Fair request processing• File placement in disk• Garbage collection

A. Sim, CRD, L B N L 4Oct. 23, 2008

Some Use Cases

• STAR experiment• Data replication between BNL and LBNL

• HPSS access at BNL and NERSC• SRMs in production for over 4 years

• Part of analysis scenario to move job-generated data files from PDSF/NERSC to remote BNL storage

• Earth System Grid• Serving about 6000 users

• Over a million files and 170TB of climate data • from 5 storage sites with SRMs (LANL, LLNL, NCAR, NERSC, ORNL)

• Uses an adapted SRM for NCAR’s own MSS

A. Sim, CRD, L B N L 5Oct. 23, 2008

• Data Replication from BNL to LBNL• SRMs in production for over 4 years

• Event processing in Grid Collector• Prototype uses SRMs and FastBit indexing embedded in STAR

framework• STAR analysis framework

• Job driven data movement1. Use BeStMan to bring files into local disk from a remote file repository2. Execute jobs that access “staged in” files in local disk3. Job creates an output file on local disk4. Job uses BeStMan to move the output file from local storage to remote

archival location5. SRM cleans up local disk when transfer complete6. Can use any other SRMs implementing v2.2

STAR experiment

A. Sim, CRD, L B N L 6Oct. 23, 2008

STAR Analysis scenario

BeStMan

Disk

Cache

DISK CACHE

Client Job Gate Node

Worker Nodes

Disk

Disk

Disk

Disk

Client Job

Client Job

Client Job

Disk

Cache

BeStMan

Disk

Cache

DiskBeStMan

Disk

DiskGridFTPserver

Disk

SRMs

ClientJob submissionRemote sites

A site

A. Sim, CRD, L B N L 7Oct. 23, 2008

BeStMan Gateway ModeExtra Slides

A. Sim, CRD, L B N L 8Oct. 23, 2008

How it works all together in Put/Get

File SystemGridftp server

Gridftp server

Gridftp server

.

.

.

BeStMan Gateway

Client

srmPrepareToGet/PutTURL

GridFTP file transfers

srmReleaseFiles/srmPutDone

A. Sim, CRD, L B N L 9Oct. 23, 2008

How it works all together in Ls/Rm/Mkdir/Rmdir

File systemGridftp server

Gridftp server

Gridftp server

.

.

.

BeStMan Gateway

Client

srmLs/srmRm/srmMkdir/srmRmdir

A. Sim, CRD, L B N L 10Oct. 23, 2008

Extra Slideson

SRM

A. Sim, CRD, L B N L 11Oct. 23, 2008

• Storage Resource Managers (SRMs) are middleware components• whose function is to provide

• dynamic space allocation and • file management

on shared storage components on the Grid• Different implementations for underlying storage

systems based on the SRM specification

What is SRM?

A. Sim, CRD, L B N L 12Oct. 23, 2008

• Get/put files from/into spaces

• archived files on mass storage systems• Shared storage space allocation & reservation

• important for data intensive applications

• File transfers from/to remote sites, file replication

• Negotiate transfer protocols

• File and space management with lifetime

• support non-blocking (asynchronous) requests

• Directory management

• Interoperate with other SRMs

SRMs in the data grid

A. Sim, CRD, L B N L 13Oct. 23, 2008

History

• 7 year of Storage Resource (SRM) Management activity• Experience with system implementations v.1.x - 2001

• MSS: HPSS (LBNL, ORNL, BNL), Enstore (Fermi), JasMINE (Jlab), Castor (CERN), MSS (NCAR), SE (RAL) …

• Disk systems: DRM(LBNL), dCache(Fermi), DPM(CERN), jSRM (Jlab), …• SRM v2.1 spec was finalized – 2003• GSM: BOF at GGF8 - June 2003• SRM v2.2 spec was finalized – May 2006• SRM v3.0 spec being discussed - 2007

A. Sim, CRD, L B N L 14Oct. 23, 2008

SRM v2.2 Interface

• Data transfer functions to get files into SRM spaces from the client's local system or from other remote storage systems, and to retrieve them• srmPrepareToGet, srmPrepareToPut, srmBringOnline, srmCopy

• Space management functions to reserve, release, and manage spaces, their types and lifetimes. • srmReserveSpace, srmReleaseSpace, srmUpdateSpace, srmGetSpaceTokens

• Lifetime management functions to manage lifetimes of space and files.• srmReleaseFiles, srmPutDone, srmExtendFileLifeTime

• Directory management functions to create/remove directories, rename files, remove files and retrieve file information.• srmMkdir, srmRmdir, srmMv, srmRm, srmLs

• Request management functions to query status of requests and manage requests• srmStatusOf{Get,Put,Copy,BringOnline}Request, srmGetRequestSummary,

srmGetRequestTokens, srmAbortRequest, srmAbortFiles, srmSuspendRequest, srmResumeRequest

• Other functions include Discovery and Permission functions• srmPing, srmGetTransferProtocols, srmCheckPermission, srmSetPermission, etc.

A. Sim, CRD, L B N L 15Oct. 23, 2008

Interoperability ?

• Storage Resource Managers (SRMs) are based on a common interface specification.• SRMs can have different implementations for the underlying

storage systems.• Compatibility and interoperability need to be tested according to

the specification.• 5 server implementations are currently available for v2.2

• CASTOR (CERN, RAL)• dCache (FNAL, DESY)• DPM (CERN)• StoRM (Italy)• BeStMan (LBNL)