RDIG (Russian Data Intensive Grid) RDIG (Russian Data Intensive Grid) e-Infrastructure: Status and Planse-Infrastructure: Status and Plans

Vyacheslav Ilyin (SINP, MSU), Vyacheslav Ilyin (SINP, MSU), Vladimir Korenkov (JINR, Dubna), Vladimir Korenkov (JINR, Dubna), Aleksey Soldatov (RRC “Kurchatov Institute”)Aleksey Soldatov (RRC “Kurchatov Institute”)

NEC’2009, Varna10 September, 2009

Some history

• 1999 – Monarc Project– Early discussions on how to organise

distributed computing for LHC• 2001-2003 - EU DataGrid project

– middleware & testbed for an operational grid

• 2002-2005 – LHC Computing Grid – LCG– deploying the results of DataGrid to provide

aproduction facility for LHC experiments

• 2004-2006 – EU EGEE project phase 1– starts from the LCG grid– shared production infrastructure– expanding to other communities and

sciences• 2006-2008 – EU EGEE-II

– Building on phase 1– Expanding applications and communities … 2008-2010 – EU EGEE-III

CERN

EGEEEGEE (Enabling Grids (Enabling Grids for E-sciencfor E-sciencEE))

The aim of the project is to create a global Pan-European computing infrastructure of a Grid type. - Integrate regional Grid efforts - Represent leading grid activities in Europe

10 Federations, 27 Countries, 70 Organizations

350 sites55 countries120,000 CPUs26 PetaBytes>15,000 users>300 VOs>370,000 jobs/day

ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…

The Russian consortium RDIG (Russian Data Intensive Grid, was set up in September 2003 as a national federation in the EGEE project.

IHEP - Institute of High Energy Physics (Protvino), IMPB RAS - Institute of Mathematical Problems in Biology (Pushchino), ITEP - Institute of Theoretical and Experimental Physics JINR - Joint Institute for Nuclear Research (Dubna), KIAM RAS - Keldysh Institute of Applied Mathematics PNPI - Petersburg Nuclear Physics Institute (Gatchina), RRC KI - Russian Research Center “Kurchatov Institute” SINP-MSU - Skobeltsyn Institute of Nuclear Physics,MSU,

The Russian consortium RDIG

Now the RDIG infrastructure comprises 15 Resource Centers with more 7000 kSI2K CPU and more 1850 TB of disc storage.

The RDIG infrastructure

• RDIG Resource Centres:– ITEP– JINR-LCG2– Kharkov-KIPT– RRC-KI– RU-Moscow-KIAM– RU-Phys-SPbSU– RU-Protvino-IHEP– RU-SPbSU– Ru-Troitsk-INR– ru-IMPB-LCG2– ru-Moscow-FIAN– ru-Moscow-GCRAS– ru-Moscow-MEPHI– ru-PNPI-LCG2– ru-Moscow-SINP

The main directions in development and maintenance of RDIG e-infrastructure

are as the following: • - support of basic grid-services;• - Support of Regional Operations Center (ROC);• - Support of Resource Centers (RC) in Russia;• - RDIG Certification Authority;• - RDIG Monitoring and Accounting;• - participation in integration, testing, certification of grid-

software; • - support of Users, Virtual Organization (VO) and application;• - User & Administrator training and education;• - Dissemination, outreach and Communication grid activities.

Development and maintenance of RDIG e-infrastructure

www.egee-rdig.ru

• Infrastructure VO's (all RC's):– dteam– ops• Most RC support the WLCG/EGEE VO's– Alice– Atlas– CMS– LHCb• Supported by some RC's:– gear– Biomed– Fusion• Regional VO's– Ams, eearth, photon, rdteam, rgstest

VOs

Flagship applications: LHC, Fusion (toward to ITER), nanotechnologyCurrent interests from: medicine, engineering

RDIG - training, induction courses

PNPI, RRC KI, JINR, IHEP, ITEP, SINP MSU:

• Induction courses ~ 600 • Courses for application developers ~ 60 • Site administrator training ~ 100

today more than 200 physicists from different LHC centers in Russia got this training

RDIG monitoring & accounting http://rocmon.jinr.ru:8080

Architecture

cluster of institutional computing centers;

major centers are of Tier2 level (RRC KI, JINR, IHEP, ITEP, PNPI, SINP MSU), other are (will be) Tier3s. each of the T2-sites operates for all four experiments - ALICE, ATLAS, CMS and LHCb.

this model assumes partition/sharing of the facilities (DISK/Tape and CPU) between experiments.

basic functions: analysis of real data; MC generation; users data support plus analysis of some portion of RAW/ESD data for tuning/developing reconstruction algorithms and corresponding programming.

Thus, approximately equal partitioning of the storage: Real AOD ~ Real RAW/ESD ~ SimData

:Russia in World-wide LHC Computing Grid

LHC experiments support

– Networking– Computer

power– Data storage– Software

installation and maintenance

– Mathematical support

Grid – solution for LHC experiments

Tier-2s and Tier-1s are inter-connected by the general

purpose research networks

Any Tier-2 mayaccess data at

any Tier-1

Tier-2 IN2P3

TRIUMF

ASCC

FNAL

BNL

Nordic

CNAF

SARAPIC

RAL

GridKa

Tier-2

Tier-2

Tier-2

Tier-2

Tier-2

Tier-2

Tier-2Tier-2

Tier-2

Tiers-1 for JINR

ATLAS, CMS,ALICE

T1s for Russia:

ALICE - FZK (Karlsruhe) to serve as a canonical T1 center for Russia T2 sites

ATLAS - SARA (Amsterdam) to serve as a canonical T1 center for Russia T2 sites

LHCb - CERN facilities to serve as a canonical T1 center for Russia T2 sites

CMS - CERN as a CMS T1 centre for the purposes of receiving Monte Carlo data from Russian T2 centres. Also as a special-purpose T1 centre for CMS, taking a share of the general distribution of AOD and RECO data as required.

Russia in World-wide LHC Computing Grid

RDIG resource planning for LHC:

Russia in World-wide LHC Computing Grid

2008 2009 2010 2011 2012 2013 ALICE ATLAS CMS LHCb

CPU (MSI2K) 3.0 6.2 8.5 12.5 16.0 24.0 26% 26% 29% 19%

Disk (PByte) 0.6 1.8 3.0 4.0 6.0 8.0

WAN (Gbit/sec) 2.5 2.5 5.0 5.0 10.0 10.0

The protocol between CERN, Russia and JINR on a participation in LCG Project has been

approved in 2003.

The tasks of the Russian institutes in the LCG have been defined as:

LCG software testing;

evaluation of new Grid technologies (e.g. Globus toolkit 3) in a context of using in the LCG;

event generators repository, data base of physical events: support and development.

LHC Computing Grid Project LHC Computing Grid Project (LCG)(LCG)

The tasks of the Russian institutes & JINR in the LCG (2009 years):

• Task 1. MW (gLite) Testsuit (supervisor O. Keeble) • Task 2. LCG vs Experiments (supervisor I. Bird)• Task 3. LCG monitoring (supervisor J. Andreeva)• Task 4/5. Genser/ MCDB ( supervisor A. Ribon)

LHC Computing Grid Project (LCG)LHC Computing Grid Project (LCG)

International connectivity for RDIG/LCG:

for connection with LCG centers in Europe:GEANT2 PoP - 2 links 2.5 Gbps (10 Gbps in

process)

dedicated connectivity to CERN: 1 Gbps connection Moscow-Amsterdam - CERN

Russia in World-wide LHC Computing Grid

Moscow 10 Gbps - RRC KI, SINP MSU (in process)

1 Gbps - ITEP, LPI, MEPhI

JINR (Dubna) 20 Gbps

IHEP (Protvino) 1 Gbps

INR RAS (Troitsk) 1 Gbps

PNPI (Gatchina) 1 Gbps

SPbSU (S-Peterburg) 1 Gbps

BINP (Novosibirsk) 100 Mbps

Russia in World-wide LHC Computing Grid

Collaboration with RSCC, RIPN, MSK-IX, JET Infosystems, Nortel

JINR - Moscow telecommunication channel

28

JINR Central Information and Computing Complex (CICC)

29

In 2008In 2008, total CICC , total CICC performance was 1400 performance was 1400 kSI2KkSI2K, , the disk storage capacity 100 TB the disk storage capacity 100 TB

At presentAt present, the CICC , the CICC performance equals performance equals 2300 2300 kSI2KkSI2K (960 core) and the disk (960 core) and the disk storage capacity 500 TB storage capacity 500 TB

2008 2009 2010

CPU(kSI2K)

1250 1750 2500

Disk (TB)

400 800 1200

Tapes(TB)

- - 500

SuperBlade – 5 BOX80 CPU Xeon 5430

2.66 GHz Quad Core(two with InfiniBand)

Production Normalised CPU time per EGEE site for VO LHC (June – September 2009)

30

GRID-site CPU time Num CPU------------------------------------------------------------------------FZK-LCG 8,095,787 8620CERN-PROD 4,552,891 6812INFN-T1 4,334,940 2862GRIF 4,089,269 3454JINR 3,957,790 960CYFRONET-LCG 3,948,857 2384PIC 3,921,569 1337UKI-GLASGOW 3,860,298 1912RAL-LCG2 3,793,504 2532UKI-LT2-IC-HEP 3,752,747 960IN2P3-CC 3,630,425 4544

Frames for Grid cooperation of JINR in 2009

• Worldwide LHC Computing Grid (WLCG);• Enabling Grids for E-sciencE (EGEE);• RDIG Development (Project of FASI)• CERN-RFBR project “GRID Monitoring from VO perspective”• BMBF grant “Development of the GRID-infrastructure and tools to

provide joint investigations performed with participation of JINR and German research centers’

• “Development of Grid segment for the LHC experiments” was supported in frames of JINR-South Africa cooperation agreement;

• NATO project "DREAMS-ASIA“ (Development of gRid EnAbling technology in Medicine&Science for Central ASIA);

• JINR-Romania cooperation Hulubei-Meshcheryakov programme • Project "SKIF-GRID" (Program of Belarussian-Russian Union State). • Project GridNNN (National Nanotechnological Net)• We work in close cooperation and provide support to our partners in

Armenia, Belarus, Bulgaria, Czech Republic, Georgia, Germany, Poland, Romania, South Africa, Ukraine, Uzbekistan.