Padova, 5 October

StoRM Service view

Riccardo Zappi INFN-CNAF

Bologna

Padova, 5 October Storage Task Force MeetingPage 2

Outline

• Forewords– SNIA – SSM Model– Storage System example – Storage Element

• What SRM and StoRM are• Aggregation within Application layer

(difference with DPM and dCache)• StoRM feasible tests• Storage resource access scenario with

StoRM

Padova, 5 October Storage Task Force MeetingPage 3

Stor

age

dom

ain

The SNIA–SSM: Shared Storage Model

Application

File/record layerDatabase(dbms)

File system(fs)

Block layerStorage devices (disks, …)

Block aggregationHost

Network

Device

Services

Discovery, monitoring

Resource mgmt, configuration

Security, billing

Redundancy mgmt (backup, …)

High availability(fail-over, …)

Capacity planning

Mgmt ClientSMIS

Padova, 5 October Storage Task Force MeetingPage 4

Device block-aggregation

Network block-aggregation

Host block-aggregation

Clustering inter-box aggregation (example)

File

/reco

rdla

yer

Blo

ckla

yer

Host

NAS head

Host

Disk array

SN

NAS head

LAN

Cluster FS

Purposes:• load spreading across peers (scalability)

• alternate paths (high availability, scalability)

Application

GPFS

Lustre

Padova, 5 October Storage Task Force MeetingPage 5

File

/reco

rdla

yer

Blo

ckla

yer

Host Host

Application

LANFilemetadata

NAS head

Files Aggregation metadata

NAS head

Aggregation within Application layer

DPM

- No cluster-FS.

dCache

- Hierarchical MSS

Padova, 5 October Storage Task Force MeetingPage 6

Grid Storage Element • Storage Element (SE) represents a storage system

abstraction in Grid.

ServiceI/O

Data Transfer

HTTPS

...

GSIFTP

ManagementData Access

RFIO

dCap

...

POSIX

PBox

Admin

...

SRM

Information

...

Local Info Provider

Storage System

File/Record Layer

Block LayerServices

Padova, 5 October Storage Task Force MeetingPage 7

SRM : Storage Mgmt Middleware within the Grid

Storage Resource Managers (SRMs)• SRMs are middleware components that manage

shared storage resources on the grid and provide management functionalities like:– Uniform access to heterogeneous types of

storage– File pinning– Disk space allocation and advanced disk space

reservation– Protocol negotiation– Life time management of files– Management of security

• SRMs implements a SRM interface specified by GSM-WG within GGF. (v.1.1, v.2.1.1 and v.3.0)

Padova, 5 October Storage Task Force MeetingPage 8

Storage System

GPFS

Storage System

LUSTRE

Storage System

DCACHEdC

Storage System

CASTOR

SRM-dCACHE

Managent Access & Data Access to SE

Access Protocol

rfioAccess Protocol

dCapAccess Protocol

posix

Data Access

Management Access

SRM-Castor StoRMManagement Protocol

SRM v.1.1Management Protocol

SRM v.2.1.1

Padova, 5 October Storage Task Force MeetingPage 9

What StoRM is

• StoRM is a Storage Resource Manager.

• It is exposed as web service.– Web service description (wsdl) of StoRM is compliant

with SRM specification version 2.1.1• StoRM prefers Cluster File System with ACL

support.• Current StoRM version is built on top of GPFS

(Plugin for Lustre FS is coming soon). • StoRM allows direct access (native posix call)

[i.e. StoRM works without server I/O too]. Aggregation is performed by underlying file system (better than StoRM could do).

Padova, 5 October Storage Task Force MeetingPage 10

Feasible tests on Grid Storage System

Storage SystemTests on underlying Storage System with

different Hardware/Software configurations.

Storage Services• Data Access by Jobs running in WNs. • Data Transfer by Transfer Service.• Management calls by jobs, services, clients.• Information provider… ( ? )

Padova, 5 October Storage Task Force MeetingPage 11

StoRM (SRM) test• StoRM is a management service. • StoRM is not a I/O service.

TESTs• Functionality tests

– Well/Bad formed function call test– SRM interface compliance test– Security hacking

• Stress tests– Bombing with well/bad formed function calls– Huge size requests

• Integration test– Use of SRM by other Service (FTS, Agreement Service, …)– Use of SRM with other Service (LCMAPS, PBox, etc, … )

• Failure test (network crash, database crash, power crash, ..) • Perverse test

Padova, 5 October Storage Task Force MeetingPage 12

StoRM test environment• Test bed #1 (V.Vagnoni) [build]

1. Only StoRM ahead of Storage System based on GPFS.– Useful for [ Functionality, Stress, Failure ] tests

• Test bed #2 (Certification INFN-Grid) [finishing]1. Storage Element (GridFTP server, LCMAPS, StoRM,

Authorization service (PBox), ecc.. )2. Computing Element and some work nodes.– Useful for [ Integration / Stress / Failure ] tests

• Test bed #3 (Certification INFN-Grid) [coming soon]1. Nr.2 (or more) Test bed #22. External Service (FTS, LFC / FireMan, … )– Useful for [ Integration / Failure / Perverse ]

Padova, 5 October Storage Task Force MeetingPage 13

Padova, 5 October

StoRM use cases

Backup splides

Padova, 5 October Storage Task Force MeetingPage 15

WN#k

GPFS

CE

Gatekeeper

JobManager

LCMAPS

LCAS

Sto

RM

Ser

ver

StoRM Service

PermissionComponent

PermissionCatalog

PBox(CE Istance)

UserJob

PrivilegeEnforcer

LCMAPS

SharedMapping

Policies & assignment

PBox(SE Istance)

GPFS File System(ACL Enforcement Mechanism)

PFNACL

SURL

SURL

Storage Element

Computing Element and WNs

Padova, 5 October Storage Task Force MeetingPage 16

References• DMTF • SNIA• SMI-S• GPFS• LUSTRE

• SRM