introduction to osg storage suchandra thapa computation institute university of chicago march 19,...

Introduction to OSG Storage

Suchandra ThapaComputation InstituteUniversity of Chicago

March 19, 2009 1GSAW 2009 Clemson

Overview• Storage in OSG

– Storage for VDT– Storage Extension– Support– Certification– Community tools– Documentation– Support– GIP & Storage Element

• Bestman – Overview– “full-mode”/“gateway”– “gateway”/Xrootd installation

• dCache– Overview– Monitoring tools– Opportunistic storage– Gratia Probes

March 19, 2009 2GSAW 2009 Clemson

Why do we care about storage?

• Sites are growing to the point where putting a bunch of disks and nfs exporting it no longer works– Too much space needed to make it economical

• LHC projects will generate several petabytes of data a year

• Typical Tier 2 sites have on the order of 500TB – 1PB of storage space

– I/O contention and loads will bring down a normal NFS server

March 19, 2009 GSAW 2009 Clemson 3

Why care about storage (the sequel)?

• Would like additional information on storage– Let users easily get space information

(available/used) without checking sites individually

• Advertise space availability and allow disk space to be reserved– If your site is sponsored by a VO like ATLAS, you

don’t want to have another VO come in and use up all the space



Solutions for OSG StorageOSG Storage for VDT is a well integrated distributed project between Wisconsin and Fermilab. Activities:

• Packaging storage software for VDT– Srm/dCache– BeStMan– BeStMan-gateway/Xrootd (NEW)

• Simplify configuration/installation for OSG• Help VOs to use storage on OSG sites• Develop and run validation tests• Develop/maintain/package accounting and monitoring tools• Support/test/package community tools• Provide users and admins support• Perform troubleshooting and debugging• OSG liaison to storage developer groups• Educate OSG community about storage, provide documentation• Participate in grid schools organized by OSG


OSG Storage Extension Project

• Tightly related to OSG Storage for VDT• Goals for this year:

– Develop software to match storage attributes of a job to a Storage Element

– Develop web interface • for SE discovery and authorization check• for creating and managing space reservations

– Work on consolidation of monitoring functionalities of Storage Elements into one interface.

– Provide electronic means for sites to announce SE maintenance period


Support Challenges

• Complicated, highly distributed services• Huge variety of configuration options (software and hardware) • Widely diverse utilization patterns • dCache is known for poor error diagnostics, exception handling and

propagation• We do not enough experience with Xrootd • Lack of monitoring/diagnostic tools• Support team does not have access to the service. Support personnel

– Often are not authorized to use the service as user– Can not access site logs and configuration– Often can not access storage monitoring pages on the site

We would like to ask storage administrators for cooperation in:• Notifying us about the reoccurring problems• Provide us access to log files, configuration files


Certification• Maintain test stands

– 6 nodes test stand for dCache– Planning to have 5 nodes test stand for BeStMan-gateway/xrootd

• Develop/run validation test suites before software is released to VDT– dCache test suite covers:

• all srm-fermi-client commands• data replication• space management• load tests

– BeSTMan testing provided by LBL• Site registration, daily test results - http://datagrid.lbl.gov/osg • Site could run tests with srm-tester-2 – instructions at

https://twiki.grid.iu.edu/twiki/bin/view/Storage/BeStMan – BeStMan-gateway/Xrootd covers:

• all supported srm-lbl-client /srm-fermi-client commands – work in progress at Fermilab

http://datagrid.lbl.gov/osg

https://twiki.grid.iu.edu/twiki/bin/view/Storage/BeStMan


Test Suites Results

Fermilab Validation Test Suite Results

LBL SRM Tester Results


Community-contributed toolkit

• Goal is to provide a boost in effectiveness and efficiency ofoperating the deployed storage

• Download from http://datagrid.ucsd.edu/toolkit

• Is packaged as rpms• Collected/packaged

by A. Rana


Storage Documentation

• Revised documentation • Main Page:

https://twiki.grid.iu.edu/twiki/bin/view/Documentation/WebHome

• Useful links under Storage Element administrators:– Opportunistic Storage/Space Reservation– Opportunistic Storage Model for USCMS– Gratia Storage Probes– Tools, Tips, FAQs– dCache Installation/references– BeStMan references


Storage Element Discovery

• Glue Schema 1.3– ControlProtocol

• SRM– AccessProtocol

• gsiftp– Storage Area

• Groups of Pools• VOInfo

– Path

• GIP/CeMon– Collects information

on CE– Sends BDII/ReSS

information– ReSS Information is

used for job matching


Storage Discovery ExampleYou can find what sites support your VO or what information is advertised by a site about SE: https://osg-ress-2.fnal.gov:8443/test/osg_storage.jsp

https://osg-ress-2.fnal.gov:8443/test/osg_storage.jsp


Berkley Storage Management (BeStMan)

• What is BeStMan?– Developed in LBNL by Scientific Data Management Research Group – Full implementation of SRM v2.2 for disk based storage systems and

mass storage systems – Supports transfer services:

• GsiFtp• Ftp• Http• Https

– Supports multiple transfer servers– GSI security with either grid-mapfile or GUMS server

• Who would benefit from BeStMan?– Sites with limited hardware resources– Sites with limited admin effort

• BeStMan could be used in two modes:– Full mode– Gateway mode


BeStMan-gateway• Generic SRM v2.2 load balancing

frontend for GridFTP servers• Light-weight implementation of SRM v2.2

for POSIX file systems– srmPing– srmLs– srmRm– srmMkdir– srmRmdir– srmPrepareToPut (Status, PutDone)– srmPrepareToGet(Status,ReleaseFiles)

• Designed to work with any Posix-like file systems– NFS, GPFS, GFS, NGFS, PNFS, HFS+, PVFS,

AFS, Lustre, XrootdFS, Hadoop

• Doesn’t support queuing or disk space management


BeStMan-gateway/Xrootd

• Xrootd - is designed to provide POSIX-like access to files and their enclosing directory namespace

• BeStMan-gateway needs the following additional components to work with Xrootd– FUSE - File System in User Space

http://fuse.sourceforge.net – XrootdFS - implements a Posix filesystem for an

Xrootd storage cluster http://wt2.slac.stanford.edu/xrootdfs/xrootdfs.html

• GridFtp needs Data Storage Interface (DSI) module in order to work with Xrootd storage

http://fuse.sourceforge.net/

http://wt2.slac.stanford.edu/xrootdfs/xrootdfs.html


BeStMan-gateway/Xrootd Architecture


Before Installing BeStMan-gateway/Xrootd

• How many nodes could be used for storage? • Minimum number of nodes is 3:• BeStMan, XroodFS, fuse, GridFtp • Xrootd redirector• Xrootd data server node

• Chose authorization mechanism do you prefer?• GUMS• gridmap-file

• Do you need to support static space tokens? • Select name for Mount Point for XrootdFS on BeStMan

node • Decide how to partition storage areas on Xrootd redirector

and data server nodes


XrootdFs Installation (I)

• Install fuse using “yum install” or rpms (e.g http://rpmfind.net/linux/rpm2html/search.php?query=fuse)– fuse-2.7.3-1– fuse-libs-2.7.3-1– kernel-module-fuse-2.6.9-78.0.1.EL-2.7.3-1

• Install and configure XrootdFSpacman -get http://vdt.cs.wisc.edu/vdt_XXX_cache/XrootdFS$VDT_LOCATION/vdt/setup/configure_xrootdfs \ --user <user> \ --cache <mount-point> \ --xrdr-host <hostname> \ --xrdr-storage-path <path>


BeStMan-gateway Installation (II)

• Install and configure BestManpacman -get OSG:Bestman $VDT_LOCATION/vdt/setup/configure_bestman --server y \ --user <user> \ --cert <service_cert> \ --key <service_key> \ --http-port <public_port> \ --https-port <secured_port> \ --gums-host <GUMS hostname> \ --gums-port <GUMS port number> \ --gums-dn <Client DN for GUMS interface> \ --use-xrootd \ --with-tokens-list "<TOKEN_1_NAME>[desc:<TOKEN_1_DESC]

[TOKEN_1_SIZE_GB];TOKEN_2_NAME[desc:TOKEN_2_DESC][TOKEN_2_SIZE]" \ --with-transfer-servers <GridFTP server list>

• Modify /etc/sudoersCmnd_Alias SRM_CMD = /bin/rm, /bin/mkdir, /bin/rmdir, /bin/mv, /bin/lsRunas_Alias SRM_USR = ALL, !root<user_name> ALL=(SRM_USR) NOPASSWD: SRM_CMD

http://vdt.cs.wisc.edu/vdt_XXX_cache/bestman


GridFtp Installation (II)

• Install and configure GridFtp pacman -get OSG:Xrootd-GridFTP$VDT_LOCATION/vdt/setup/configure_gridftp --use-xrootd \ --xrootd-host <hostname> \--xrootd-mount-point <mount_point> \ --xrootd-storage-path < path>

http://vdt.cs.wisc.edu/vdt_XXX_cache:Xrootd-GridFTP


Xrootd-Redirector Installation (IV)

• Install and configure Xrootd redirector pacman -get OSG:Xrootd $VDT_LOCATION/vdt/setup/configure_xrootd \--server y \--this-is-xrdr \--user <user> \--xrdr-storage-path <path> \--xrdr-storage-cache <cache> \--with-tokens-list "<TOKEN_1_NAME>[desc:<TOKEN_1_DESC]

[TOKEN_1_SIZE_GB];TOKEN_2_NAME[desc:TOKEN_2_DESC][TOKEN_2_SIZE]" \--public-cache-size <PUBLIC_SPACE_SIZE>


Xrootd Data Server Installation

(V)• Install and configure Xrootd data server

pacman -get OSG:rootd$VDT_LOCATION/vdt/setup/configure_xrootd \--server y \--user <user> \--xrdr-host <hostname> \--xrdr-storage-path <path> \ --xrdr-storage-cache <cache> \ --with-tokens-list "<TOKEN_1_NAME>[desc:<TOKEN_1_DESC]

[TOKEN_1_SIZE_GB];TOKEN_2_NAME[desc:TOKEN_2_DESC][TOKEN_2_SIZE]" \--public-cache-size <PUBLIC_SPACE_SIZE>


Start/stop order for BeStMan-gateway/Xrootd

• Use vdt-control –on/--off mechanism• Start severs in the following order (use

reverse order to stop):– Xrootd redirector– Xrootd data server– GridFtp– XrootdFs/Bestman-gateway

• Testsrm-ping srm://<BeStMan_host>:8443/srm/v2/serversrm-copy file:////tmp/test srm://<BeStMan_host>.gov:8443/srm/v2/server\?

SFN=<MOUNT_POINT>/test -spacetoken <TOKEN_1_NAME>


• BeStMan is one of data storage solutions supported by OSG. BeStMan-gateway, BeStMan-gateway/Xrootd requested by ATLAS are just released in VDT

• Current version of software– BeStMan - 2.2.1.2.e1 – XrootdFS - 2.2.1.1 – GridFTP-Xrootd ,xrootd-dsi-20080828-1632 – Prima 0.7.1– Xrootd - 20080828-1632

• VDT configuration script tailored to set up BeStMan “full mode”/gateway for Tier-2/Tier-3

• BeStMan srm-clients are distributed as a part of VDT client cache– Fermi client– LBNL client– LCG-utils

• There are several installations of BeStMan on OSG sites

BeStMan in VDT


dCache Main Features• nfs-mountable namespace• Multiple access protocols

– dcap (posix io), gsidcap– xrootd (posix io)– gsiftp (multiple channels)

• Replica Manager– Set min/max number of replicas

• Role-based authorization– Selection of authorization mechanisms

• Opportunistic storage• Billing• Admin interface

– ssh, jython• InformationProvider (not in production - yet)

– SRM and gsiftp described in glue schema• Platform, fs independent (Java)

– 32 and 64-bit linux, solaris; ext3, xfs, zfs


dCache OSG Tier-2 site Architecture

Slide courtesy of Ted Hesselroth (from presentation: “Installing and Using SRM-dCache”)


dCache MonitoringNumerous ways to

monitordCache• Info Pages -

http://localhost:2288• SrmWatch -

http://localhost:8080/srmwatch/

• Admin Interface ( UI or from command line)

• Scripts/web pages built by community – http://fndca.fnal.gov – http://fndca2a.fnal.gov:9090/lps/plots/

src/plots.lzx


SRMWatch Examples


Gratia ServiceGratia is the accounting service for OSG is provided by the

Gratia external project. • Main goal is to provide the stakeholders with a reliable and

accurate set of views of the Grid resources usage. • Job and other accounting information gathered by Gratia

probes run on the compute element or other site nodes are reported to a Gratia collectors – Fermi collector: http://gratia-fermi.fnal.gov:8886/gratia-

reporting – OSG collector: http://gratia.opensciencegrid.org:8886/gratia-

reporting • Accounting records collected by Gratia are forwarded to the

EGEE accounting system, APEL:– http://www3.egee.cesga.es/gridsite/accounting/CESGA/

osg_view.html


dCache Gratia Probes• dCache Gratia Probes

– Storage Probe – Transfer Probe

• Storage Probe – Is responsible for reporting storage capacity and storage usage– Gets the pool information from the dCache admin server– Gets the SRM information from the SRM tables in the SRM Postgres

database– Runs as a cron job on the host running the Postgres database server

for SRM• Transfer Probe

– Reports the details of each file transfer into or out of a dCache file server

– Gets this information from the dCache "billing" database. – Runs as a daemon process– For performance reasons, sites with large dCache billing databases

are advised to alter the "billinginfo" table by indexing specific tables in order speed up the search for newly added records


Gratia Transfer Probes Report


dCache Information Service• A snapshot of the current status of a dCache instance for external

consumption– Works independently of the rest of dCache.– Pools information for various dCache components (configurable

update period).• Doesn't provide historic data• Not an info-provider but GIP is a customer• Access data:

– Via the admin interface– Via XML Conduit - A TCP connection get complete state as XML.– Web front-end– Supports some advanced features:

• Derived data (re-)calculated as state changes.• Multiple output formats and transports.• If additional metrics, data formats or transports are needed, they

can be added


From Paul Millar’s presentation: “dCache seminar: introducing the info service “http://www.dcache.org/manuals/dCache-info-20080813.pdf

Information Service Architecture


Opportunistic Storage• Opportunistic Storage in dCache 1.8 with SRM 2.2

– Provides a capability of specifying a portion of the total storage for opportunistic use

– Allows particular VOs and Roles a privilege to use space other than that included in opportunistic storage

– Files created through opportunistic use will not be permanently available in the storage system

– A storage site administrator may configure the site for opportunistic use through space reservation.

– Creation of space reservations is controlled by use of link groups– The administrator may assign storage pools to link groups– Certain pools are designated for opportunistic use.

• Numerous documents describing how to install and operate Opportunistic Storgae on Tier-2 sites– https://twiki.grid.iu.edu/twiki/bin/view/Storage/

OpportunisticStorageSetup– https://twiki.grid.iu.edu/bin/view/Storage/

OpportunisticStorageModelForCMS


dCache in VDT• dCache is one of the data storage solutions supported by

OSG• dCache could be installed from VDT • Current version is vdt 2.2.8 (dcache 1.8.0.15 p11)• Distribution contains dCache-server, pnfs, postgress, gratia

probes rpms and a configuration script tailored to set up dCache for Tier-2/Tier-3

• Configuration script allows to do system setup, enable opportunistic storage, replication etc

• dCache-clients are distributed as a part of VDT client cache– Fermi client– LBNL client– LCG-utils

• There are multiple installations of dCache on OSG sites


Summary

• We will continue to work on improving storage packaging in VDT– Feedback is welcome!

• We are trying to make support more efficient by providing FQA, debugging the most frequently occurred problems, working with developers on improving logging and error diagnostic– The quality of the support depends greatly on Storage Admins cooperation!!!

• We will add BeStMan test stand and will do BeStMan/Xrootd certification the same way it is done with dCache release

• We will be glad to accept/package more community tools• We are trying to maintain documentation up-to-date, adding new

interesting references and “how to do” tips– Please let us know if we are missing some important topics!

• As a liaison to software developers we will be happy to pass your requests/suggestions

Acknowledgements

• Tanya Levshina


introduction to osg storage suchandra thapa computation institute university of chicago march 19,...

Documents

storage attributes

storage administrators

osg storageosg storage

packaging storage software

storage monitoring pages

osg sitesdevelop

space information

disk space