optimization of the job submission and data access in a lhc tier2 · 2011-04-14 · jobs may read...

Optimization of the job submission

and data access in a LHC Tier2

SPINOSO VincenzoEGI User Forum Vilnius, 11-14 April 2011

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 2/28

Overview● User needs● Administration issues● INFN Bari farm design and deployment

● Storage access optimization● File system performance● Performance over the WAN link

● Interactive jobs


User needs● Grid submission

● Local submission

● Interactive facilities● Code development, debugging● Analysis with ROOT● Personal research data● Backups● Editing

● Efficient I/O when serving analysis jobs● Jobs may read from storage at 12MB/s

● Fast and reliable WAN transfers (SRM, GridFTP, Xrootd)


Admin issues● Improving reliability and efficiency of the

services provided● Sharing and consolidation to avoid duplication of

services

● Support to different VOs● Support to different use cases● Support to different types of access (grid, local,

interactive)


Farm layout


Storage access● Lustre

● POSIX parallel file system

● StoRM● SRM layer on top of Lustre (CMS)

● Xrootd● Alice production instance● CMS test instance

● Different storage brands● Different technologies (HW/SW RAIDs, RAID 5/6, FC,

external SAS)


Storage pre-production● Lustre

● 5 disk servers

● Network 4x 1Gbps each

● 190 TB

● ~600 concurrent jobs

● Result: 400MB/s RW

400MB/s


Storage in production● CMS job robot efficiency is 95%


Storage in production● 250TB used

● 10 servers

● 800 concurrent jobs

● Real ROOT analysis

● Result: up to 1.3GB/s (max)

1Gbps1Gbps


Storage in production● 500TB in production

● 15 servers

● Real user activity

● Result: concurrent reads up to 2GB/s (max)


Storage in production● 650 TB in production

● 20 servers

● Real user activity

● Result: concurrent reads up to 2Gbps on average

2Gbps


CMS feedback from the grid● IO performance tests (L. Sala)

● CMS walltime for the job● CMSSW_CpuPercentage (UserTime/WallTime)

● Feedback● CPU efficiency highly improved● total execution time decreased


2Gbps

2Gbps

2Gbps WAN link

BARIBARI


Download from T1/T2

173 MB/s173 MB/s


Download from T2

145 MB/s145 MB/s


Upload to FNAL

BARI → FNAL: 237 MB/s


Xrootd tests● Running ~50 jobs at TRIESTE (CMS T3)● Jobs are reading data stored at Bari (remote

access using XRootd)

BARI → TRIESTEBARI → TRIESTE1Gbps spikes1Gbps spikes


Interactive jobs: why● Classic interactive cluster issues

● maintenance issue (ad-hoc configuration, consistency)

● scalability

● performance degradation on heavy load

● different requirements by different use cases (even if coming from the same VO)

● Interactive access through interactive jobs● The interactive submission is similar to the batch submission: the batch manager

chooses one CPU to execute the job and returns an interactive shell

● The user will keep that CPU until releasing the interactive job (logout)

● Maintenance: one unique cluster provides both batch and interactive access; the environment is the same, no consistency issues

● Scalability: the interactive cluster can increase in size, dinamically, depending on the user requests

● Performance: one CPU per user, so the users are never sharing the same core


Interactive jobs: how● Interactive jobs are provided by Torque as a functionality

● LFS has it as well

● The maui configuration is tuned a bit in order to guarantee high priority to those jobs

● A simple (custom) daemon guarantees the user that he will wait at most 60 seconds to get interactive access

● Interactive jobs can be “logged out” and “re-logged in” afterwards, using “screen”

● No hard limit on number of concurrent “interactive sessions”

● You can run also multiCPU interactive jobs● User can ask for n nodes, m processors per node


Interactive jobs AND the file system

● GOAL: we wanted one file system both for user and global data, for all the VOs on the site● the file system had to be fast and POSIX compliant, in order to support

interactive sessions just like a local filesystem● the file system had to be shared on all the nodes of the farm, in order

to allow both batch and interactive jobs to access both the user home directories and the globally available data stored on site

● needed a solution which allowed a warm upgrade of the disk space

● CHOICE: POSIX high performance cluster file system was preferred: Lustre.

● StoRM on top of Lustre to provide the SRM service


Interactive jobs example

1. Access to the frontend

2. Get a CPU

3. Use the CPU

4. Release the CPU

5. Release the frontend shell


Interactive jobs example


People involved

● Donvito Giacinto – INFN, Università di Bari● Spinoso Vincenzo – INFN, Università di Bari● Maggi Giorgio Pietro – INFN, Politecnico di Bari


References● Lustre Wiki

● http://wiki.lustre.org/index.php/Main_Page

● StoRM● http://storm.forge.cnaf.infn.it

● Xrootd● http://xrootd.slac.stanford.edu/

● Interactive jobs using qsub● http://www.clusterresources.com

http://wiki.lustre.org/index.php/Main_Page

http://storm.forge.cnaf.infn.it/

http://xrootd.slac.stanford.edu/

http://www.clusterresources.com/

optimization of the job submission and data access in a lhc tier2 · 2011-04-14 · jobs may read...

Documents