StoRMSome basics and a comparison

with DPMWahid Bhimji

University of EdinburghGridPP Storage Workshop

31-Mar-10 1Wahid Bhimji – StoRM

This talk:

Considering moving to StoRM ???– Comparison with DPM– Benefits / Problems – Status

31-Mar-10 Wahid Bhimji – StoRM 2

Whats StoRM?

• Lightweight SRM server• Works on top of POSIX filesystems• Relies on that filesystem to replicate data etc.• Used widely in Italy and a few place outside

31-Mar-10 Wahid Bhimji – StoRM 3

DPM - reminder

31-Mar-10 Wahid Bhimji – StoRM 4

DPMHead Node

PoolNode

Head Node (dpm-mysql)- SRMV2 Interface

- DPM server: Request tracking (MySQL)

- DPM Nameserver (DPNS) (MySQL)

User

WorkerNode DISK

“Pool” node (dpm-disk)

- Directly mounted disk

- RFIO– local transfers

- GridFTP– WAN transfers

…

PoolNode DISK

PoolNode DISK

gridftp

rfio

…

StoRM: Can be one machine.

Storm Structure

31-Mar-10 Wahid Bhimji – StoRM 5

Storm Frontend

Storm Backend

Frontend:SRM Interface

Request Tracking (MySQL)

Backend:Namespace (by file path)

Filesytem Drivers

Disk ServersDisk

ServersDisk ServersDisk

ServersDisk Servers

User

Worker Node file://

GridFTP server(s)GridFTP

server(s)

Differences to DPM

• Namespace derived from filename• Local transfers through file:// POSIX access• Uses ACLs for permissions • Lot less (no) user tools

– So no equivalent of dpm-* dpns-* tools– But filesystem operations are just du * rm * ls * – For spacetoken allocations etc. edit file:

• /opt/storm/backend/etc/namespace.xml

31-Mar-10 Wahid Bhimji – StoRM 6

Namespace

srm://se2.glite.ecdf.ed.ac.uk:8444/srm/managerv2?=/atlas/atlasdatadisk/bill/bob

DPM this would be DPNS path and to get to physical file you would need to use DB

- On Storm there is no DB. The physical file is at: SPACE_ROOT/bill/bobSPACE_ROOT is defined for each VO/ spacetoken: Might be just /lustre/atlas/atlasdatadisk

31-Mar-10 Wahid Bhimji – StoRM 7

Versions

• Storm 1.5 released recently • 1.4 no longer “supported”• Currently for SL4 (32bit – prob. works on 64)• SL5 – “after easter”• Installations at QMUL and Edinburgh

– both have issues at the moment– probably resolvable.

• INFN-T1 are using 1.5 most other sites 1.4

31-Mar-10 Wahid Bhimji – StoRM 8

Instalation

• Obtain reposhttp://grid-it.cnaf.infn.it/mrepo/repos/glite-generic.repo ; lcg-ca.repo; ig.repo; jpackage.repo• “Standard” yum install / YAIM configure :yum install lcg-CA ig_SE_storm_backend ig_SE_storm_frontend

/opt/glite/yaim/bin/ig_yaim -c -s siteinfo/site-info.def –n ig_SE_storm_backend -n ig_SE_storm_frontend

• Setup filesystem (Lustre/GPFS etc.; acls needed)• You also need users etc (as for DPM + other nodes)

31-Mar-10 Wahid Bhimji – StoRM 9

Weighing it up

31-Mar-10 Wahid Bhimji – StoRM 10

Benefits

Easiest current way to make use of Lustre / GPFS or other cluster filesystem:

• file:// POSIX access – better local access performance that rfio (afternoon discussion)

• Scalability / resilience (INFN-T1 uses Storm)• Simplicity: Navigable directories/ familiar

posix commands (but you also have to maintain Lustre / GPFS if you use those)

31-Mar-10 Wahid Bhimji – StoRM 11

Key reasons not to move

• Limited UK experience, smaller user community than DPM /dCache – mostly Italian sites

• Some implementation of SRM features lacking (in 1.4 – not clear if fixed in storm 1.5)– Access permissions– Information provider– Space reservation? ;-) : http://www.youtube.com/watch?v=A7uvttu8ct0

31-Mar-10 Wahid Bhimji – StoRM 12

StoRM space reservations? ;-)

31-Mar-10 Wahid Bhimji – StoRM 13

Conclusions

• StoRM is not (so) scary. – It has some quirks/limits but so does DPM– You can use the same hardware– SRM performance looks to be similar to DPM– Growing maturity and user community

• It is currently the easiest way to use cluster filesystems to give scalability and the performance of local file:// access

• Discuss (after Chris’s talk…)31-Mar-10 Wahid Bhimji – StoRM 14