srb and irods @ cc-in2p3

SRB and iRODS @ CC-IN2P3 SRB and iRODS @ CC-IN2P3

Jean-Yves Nief

Pascal Calvat

19/04/23SRB and iRODS @ CC-IN2P3 2

Overview Overview

SRB:– Status.– A couple of examples.– Prospects.

iRODS:– Developments.– Test beds.– Project in production: digital library.

Moving from SRB to iRODS ? JUX (Java Universal eXplorer): GUI in Java for many

protocols like iRODS, SRB.


SRB in LyonSRB in Lyon

HEP

BaBar SLAC « mirror » site

CMOS, Calice Data archival

Indra Data distribution and archival

Lattice QCD tens of TB / y

Astroparticle

Antares Tier 0: ~200 TB / y

Auger Tier 0: ~10 TB / y

Edelweiss Tier 0: tens of TB / y

SN Factory One of the online brick: ~GB / d

BiomedicalBioEmergence Tier 0:european project ~ TB/y

Mammography Project with a computing lab

Neuroscience Lyon and Strasbourg hospital


SRB and iRODS hardware and SRB and iRODS hardware and software @ CC-IN2P3software @ CC-IN2P3

14 SRB servers (130 TB of disk space):– Sun v480, v20z, v440 (Sparc III) and Thumpers x4500 (AMD

Opteron).– OS: Solaris 9 and Solaris 10.

Almost all the instances on SRB v3.5.0. Interfaced with our MSS: HPSS v5:

– Scripts developed to handle automatic migration/purge of compound resources (disk cache/tape archive).

MCATs: using Oracle 10g. Going to add RHEL4 machines (Linux) for MCATs enabled

servers.


Example in HEP: BaBarExample in HEP: BaBar

Data import from SLAC to Lyon. SRB being used since 2004 in production. Fully automated:

– New files created are registered in the SLAC catalog database.– Client application in Lyon: detection of files missing in the Lyon

catalog database + transfer of these files.– Automated error recovery.

Up to 5 TB / day (max. rate observed). Usual rate: 2-3 TB / day (during production periods) 700 TB imported so far (since 2004), 700,000 files. Until end 2008, volume x2.


Example in HEP: BaBarExample in HEP: BaBar

HPSS/Lyon

SRB

SRB

SRB

SRBMCAT

(1)

(3)

(2)

SRB

SRBMCAT

HPSS/SLAC

SRB

SLAC zoneSLAC zone CC-IN2P3 zoneCC-IN2P3 zone

Import agentsImport agents

Import agentsImport agents


Examples in astrophysics and Examples in astrophysics and astroparticles astroparticles

Underwater: Antares in the pampa: Pierre Auger Observatory

At the top of the mountain: SuperNovae Factory in Hawaii


Example in biology: BioEmergenceExample in biology: BioEmergence

European projects involving 5 countries.

Embryogenesis: zebra fish. 2 microscopes now (several in

the future): amount of data could be huged (PB scale).

Data pushed from the microscopes into the SRB.

SRB integrated within their workflow.

CC-IN2P3: core of the system.


VirgoVirgo

Interferometer for gravitational waves detection (in production: 60 TB / y).

Need for a reliable data distribution system.

Distribute Ligo data (same experiment in the US) to the european sites: CC-IN2P3 and Bologna.

Have been using bbftp so far.

SRB has replaced bbftp:– Bookkeeping system.– Interface with HPSS.– Handling of Ligo small files.

LIGOLIGO CC-IN2P3CC-IN2P3

CascinaCascina

BolognaBologna

Merge Ligo small filesMerge Ligo small filesduring data importduring data importon the CC-IN2P3 on the CC-IN2P3 server side server side

Push raw data to CC-Push raw data to CC-IN2P3IN2P3

Pull Virgo + Ligo Pull Virgo + Ligo data from Lyondata from Lyon

Visualize data on the Visualize data on the WAN through SRBWAN through SRB


SRB @ CC-IN2P3 SRB @ CC-IN2P3

MCATs performance enhancement:– Reindexing made automatically on a weekly basis.

Issues with Oracle performances:– Some oddity in the way Oracle optimized requests.– Request analyzis done on all the MCATs on a daily

basis.

Not completly resolved.

SRB: big consumer of Oracle resources.– Need to separate Oracle instance for the MCATs.


SRB prospects @ CC-IN2P3SRB prospects @ CC-IN2P3

More than 10 projects using it:– SRB: critical part of their computing system.– Clients on Linux, Mac OSX, Windows, Solaris, AIX (Blue Gene)

from Europe, USA. Daily traffic can be quite big:

– Hundreds of thousands of connections per day.– Some projects with more than 200,000 connections per day at

peak rate.– Bandwidth peak rate: several Gbits/s.

Will reach 1 PB of data referenced and handled by SRB in 2008.

Need to strengthen the service (add extra MCAT servers).


iRODS developments @ CC-IN2P3iRODS developments @ CC-IN2P3

Tests scripts (JY + Thomas):– icommand test script: extensive test of the binary

commands in order to track bugs (both client and server side problems).

To do: include all the new options and new icommands.

– Loading test of the system: launch n test scripts in // on the same machine.

To do: ability to launch process on other servers ? (but too specific to a given site: passwrodless ssh etc…).



Host based access control micro-service:– User id, group id.– Hostname or range of IP adresses.– Improved firewall implemented at the iRODS

level.



Load balancing micro-services (JY + Jean Aoustet):

Load balancing system:– At a given time, choose the least loaded server to

put/get data or do any other operation.– Have to be used wisely:

• It only makes sense for identical servers on the same site!– Gather metrics on each server (CPU load, network

activity, memory usage, swap, disk occupancy etc…).– Store into iCAT.– Can also be used for monitoring purposes.


Load balancing systemLoad balancing system

iRODS iCAT serveriRODS iCAT server

iRODS data serveriRODS data server




1.1. Ask each server for its Ask each server for its metrics: rule engine metrics: rule engine cron task (msi).cron task (msi).

2. Performance script 2. Performance script launched on each launched on each server.server.

Perf scriptPerf script




3. Results sent back to 3. Results sent back to the iCAT.the iCAT.

4. Store metrics into 4. Store metrics into iCAT.iCAT.

DBDB

5. Compute a «quality 5. Compute a «quality factor» for each server factor» for each server stored in an other table: stored in an other table: r.e. cron task (msi).r.e. cron task (msi).


Future developmentsFuture developments

HPSS driver: assessment by Thomas Kachelhoffer (HPSS v6) in progress.

Handling MSS resources with the protocol of your choice:– RFIO, pftp etc…– Easy way to plugin a MSS without writing a driver.– Kind of universal driver.

Micro-services for:– Handling accentuated letters in files and directory

names registered into iRODS (eg: Windows world).– Anonymization of DICOM files in biology.


iRODS test bedsiRODS test beds

With KEK (see Yoshimi talk): data transfer. LSST:

– Telescope in Chile: first light in 2014.

– Big challenge for data processing.

– USA + Chile for now. France starting to get involved.

– Computing: data management studies (DC 2 and 3).

NCSANCSA

CC-IN2P3CC-IN2P3

SDSCSDSC

(now also Indiana and Chile)(now also Indiana and Chile)


iRODS first production instanceiRODS first production instance

Just starting. Digital preservation area:

– Scan old documents and put them into iRODS.– 4-5 users at the moment: transparent access to the documents

from various platforms.– ~ 1-2 TB expected for this year.

Needs:– Propagation of ACLs to newly created collections.– Windows support for the icommands.– GUI interface (standalone or Web).– iRODS fits their needs.


Moving from SRB to iRODS ?Moving from SRB to iRODS ?

No plans yet. Will slowly moved the existing SRB projects to the

new system starting in 2010 or a bit earlier. Needs to be confident that all the bits and pieces

are there before the switch:– MSS integration: one of the key thing.

Probably brand new projects could start directly onto iRODS in 2008 such as Digilib.


Future plans for iRODSFuture plans for iRODS

Continue developments:– Focus on MSS integration.– Load balancing.

CC-IN2P3 starting to be involved with Arts and Humanities projects:– Fedora and iRODS interface to be studied.

New projects to start directly on iRODS:– 1 more candidate this year.


JUX: a Java explorerJUX: a Java explorer

Java Universal eXplorer: developed by Pascal Calvat (CC-IN2P3).

Based on JSaga APIs:– Saga (Simple Apis for Grid Application): OGF specs. Common set

of APIs to interact with middleware (storage, job tasks).– JSaga: implementation of Saga by Sylvain Reynaud, CC-IN2P3.– Plugins developped for various protocols like SRB and iRODS

(using Jargon). JUX can connect to various protocols providing the plugin

is available. Still under development: first version released soon.


AcknowledgementAcknowledgement

Thanks to:– Pascal Calvat (JUX), Jean Aoustet (Load Balancing),

Thomas Kachelhoffer (tests + HPSS)..– Wilko Kroeger (SLAC): BaBar.– Yoshimi Iida (transfer tests + iCAT stress tests).

Links: – JSaga (Sylvain Reynaud): http://grid.in2p3.fr/jsaga

http://grid.in2p3.fr/jsaga

srb and irods @ cc-in2p3

Documents

ccin2p3overview srb

ccin2p3 server

irods hardware

ccin2p3srb prospects

ccin2p314 srb servers

ccin2p3pull virgo ligo

pb of data

lyonvisualize data