optimization of the job submission and data access in a lhc tier2 · 2011-04-14 · jobs may read...

Optimization of the job submission

and data access in a LHC Tier2

SPINOSO VincenzoEGI User Forum Vilnius, 11-14 April 2011

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 2/28

Overview● User needs● Administration issues● INFN Bari farm design and deployment

● Storage access optimization● File system performance● Performance over the WAN link

● Interactive jobs

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 3/28

Overview● User needs● Administration issues● INFN Bari farm design and deployment

● Storage access optimization● File system performance● Performance over the WAN link

● Interactive jobs

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 4/28

User needs● Grid submission

● Local submission

● Interactive facilities● Code development, debugging● Analysis with ROOT● Personal research data● Backups● Editing

● Efficient I/O when serving analysis jobs● Jobs may read from storage at 12MB/s

● Fast and reliable WAN transfers (SRM, GridFTP, Xrootd)

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 5/28

Admin issues● Improving reliability and efficiency of the

services provided● Sharing and consolidation to avoid duplication of

services

● Support to different VOs● Support to different use cases● Support to different types of access (grid, local,

interactive)

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 6/28

Farm layout

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 7/28

Overview● User needs● Administration issues● INFN Bari farm design and deployment

● Storage access optimization● File system performance● Performance over the WAN link

● Interactive jobs

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 8/28

Storage access● Lustre

● POSIX parallel file system

● StoRM● SRM layer on top of Lustre (CMS)

● Xrootd● Alice production instance● CMS test instance

● Different storage brands● Different technologies (HW/SW RAIDs, RAID 5/6, FC,

external SAS)

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 9/28

Storage pre-production● Lustre

● 5 disk servers

● Network 4x 1Gbps each

● 190 TB

● ~600 concurrent jobs

● Result: 400MB/s RW

400MB/s

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 10/28

Storage in production● CMS job robot efficiency is 95%

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 11/28

Storage in production● 250TB used

● 10 servers

● 800 concurrent jobs

● Real ROOT analysis

● Result: up to 1.3GB/s (max)

1Gbps1Gbps

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 12/28

Storage in production● 500TB in production

● 15 servers

● Real user activity

● Result: concurrent reads up to 2GB/s (max)

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 13/28

Storage in production● 650 TB in production

● 20 servers

● Real user activity

● Result: concurrent reads up to 2Gbps on average

2Gbps

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 14/28

CMS feedback from the grid● IO performance tests (L. Sala)

● CMS walltime for the job● CMSSW_CpuPercentage (UserTime/WallTime)

● Feedback● CPU efficiency highly improved● total execution time decreased

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 15/28

Overview● User needs● Administration issues● INFN Bari farm design and deployment

● Storage access optimization● File system performance● Performance over the WAN link

● Interactive jobs

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 16/28

2Gbps

2Gbps

2Gbps WAN link

BARIBARI

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 17/28

Download from T1/T2

173 MB/s173 MB/s

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 18/28

Download from T2

145 MB/s145 MB/s

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 19/28

Upload to FNAL

BARI → FNAL: 237 MB/s

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 20/28

Xrootd tests● Running ~50 jobs at TRIESTE (CMS T3)● Jobs are reading data stored at Bari (remote

access using XRootd)

BARI → TRIESTEBARI → TRIESTE1Gbps spikes1Gbps spikes

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 21/28

Overview● User needs● Administration issues● INFN Bari farm design and deployment

● Storage access optimization● File system performance● Performance over the WAN link

● Interactive jobs

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 22/28

Interactive jobs: why● Classic interactive cluster issues

● maintenance issue (ad-hoc configuration, consistency)

● scalability

● performance degradation on heavy load

● different requirements by different use cases (even if coming from the same VO)

● Interactive access through interactive jobs● The interactive submission is similar to the batch submission: the batch manager

chooses one CPU to execute the job and returns an interactive shell

● The user will keep that CPU until releasing the interactive job (logout)

● Maintenance: one unique cluster provides both batch and interactive access; the environment is the same, no consistency issues

● Scalability: the interactive cluster can increase in size, dinamically, depending on the user requests

● Performance: one CPU per user, so the users are never sharing the same core

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 23/28

Interactive jobs: how● Interactive jobs are provided by Torque as a functionality

● LFS has it as well

● The maui configuration is tuned a bit in order to guarantee high priority to those jobs

● A simple (custom) daemon guarantees the user that he will wait at most 60 seconds to get interactive access

● Interactive jobs can be “logged out” and “re-logged in” afterwards, using “screen”

● No hard limit on number of concurrent “interactive sessions”

● You can run also multiCPU interactive jobs● User can ask for n nodes, m processors per node

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 24/28

Interactive jobs AND the file system

● GOAL: we wanted one file system both for user and global data, for all the VOs on the site● the file system had to be fast and POSIX compliant, in order to support

interactive sessions just like a local filesystem● the file system had to be shared on all the nodes of the farm, in order

to allow both batch and interactive jobs to access both the user home directories and the globally available data stored on site

● needed a solution which allowed a warm upgrade of the disk space

● CHOICE: POSIX high performance cluster file system was preferred: Lustre.

● StoRM on top of Lustre to provide the SRM service

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 25/28

Interactive jobs example

1. Access to the frontend

2. Get a CPU

3. Use the CPU

4. Release the CPU

5. Release the frontend shell

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 26/28

Interactive jobs example

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 27/28

People involved

● Donvito Giacinto – INFN, Università di Bari● Spinoso Vincenzo – INFN, Università di Bari● Maggi Giorgio Pietro – INFN, Politecnico di Bari

EGI User Forum – Vilnius, 11-14 April 2011 SPINOSO V. - Optimization of the job submission and data access in a LHC Tier2 28/28

References● Lustre Wiki

● http://wiki.lustre.org/index.php/Main_Page

● StoRM● http://storm.forge.cnaf.infn.it

● Xrootd● http://xrootd.slac.stanford.edu/

● Interactive jobs using qsub● http://www.clusterresources.com