kolkata tier-2@alice grid site name :- in-dae-vecc-02monalisa name:- kolkata-cream vo :- alicecity:-...

KOLKATA Tier-2@Alice Grid

Site Name :- IN-DAE-VECC-02 Monalisa Name:- Kolkata-CreamVO :- ALICE City:- KOLKATA Country :- INDIA

Shown many data transfers (few are simultaneously going on)

Kolkata Tier-2 and Network


level 0 - special hardware8 kHz (160 GB/sec)

level 1 - embedded processors

level 2 - PCs

200 Hz (4 GB/sec)

30 Hz (2.5 GB/sec)

30 Hz

(4 GB/sec)

data recording &

offline analysis

Total weight 10,000tOverall diameter 16.00mOverall length 25mMagnetic Field 0.4Tesla

ALICE Collaboration ~ 1/2 ATLAS, CMS, ~ 2x LHCb ~1100 people 30 countries, 80 Institutes

Introduction toALICE

Lots of data to Process:-Solution

GRID COMPUTINGSlide from Latchezar Betev PPT


Tier 0

Tier 18 centres

Tier 2

Different Universities, Institutes

Individual scientist’s PC

Experimental site

CERN computer centre,Geneva

ASIA(KISTI)

India China JAPAN Pakistan

FranceItalyGermanyTAIWAN

EHEPAG(VECC)

Jammu U. Panjab U.

IN_DAE_VECC_02

ALICE Grid Structure

Central Analysis Facility

CAF (T1 +T2)

Jaipur U.

10 -100 Gbps links

Logical, geographical association of regional groups

Tier 3

1 -10 Gbps links

100 -1000 Mbps links


Kolkata Tier-2 at Monalisa

Vikas Singhal, VECC, INDIA

Consistently 800 ALICE Jobs are running.



Grid Computing Facility



Total :- 888 Cores DELL Quad Core Blades 38 * 2 * 4 *2 = 608 (HT) HP Quad Core Blades 8 * 2 * 4 * 2 = 128 (HT) HP Dual Core Blades 38 * 4 = 156Equivalent 9K HEPSpec 2006 Computing Resources.Storage :- 146 TB XrootD Redirector 3 Disk Servers SAN Based

78TB (IBM) + (36 + 32)TB (HP EVA)1Gbps WAN Network speed.10Gbps Backbone Network

3 M ALICE Job completed Successfully during last 5 Years


7

NKN @ INDIA


8Dec. 23, 2009April 22, 2009 Developing e-inftrastructures in India

NKN TOPOLOGY


TIFRMumbai

CERN LHCONE

POP in Mumbai

4 Gbps/10 Gpbs Shared

TEIN-3Network

10 Gbps

10 Gbps

10 Gbps (Under going)1 Gbps

POP in Kolkata

10 Gbps

VECCKolkata

SINPKolkata

1 Gbps

KolkataTier-2

1 Gbps

Physical Network Connectivity upto Kolkata

Internet

10 Gbps

4-10 Gbps

1 Gbps

x Gbps

Core Network

10 Gbps

10 Gbps

1 Gbps

1 Gbps


NKN Router at VECC

Grid Computing Facility

1Gb optical Fiber Link

XrootD Redirector

gridce02DNS

ServerGrid-peer grid

ALICETier-2

Worker nodes

Tier-3 Nodes

ALICE Centre

1 Gigabit Ethernet Link

XrootDDisk Servers

HPIBM SAN

Storage

10 GigabitFiber Backbone

10 Gigabit Fiber Link10 Gigabit Fiber Link



Kolkata TIER-2 centre Network diagram

Router

Switch

gridce02 Backup-server

wn045

wn046

wn122

wn123

Switch-2

Internet1Gbps Copper

Computing Nodes

25 NodesDell andWipro BladesCluster with25 TB of As Tier-3

192.

168.

x.x

(Sta

nd b

y)

144.16.112.xx/27

130 TBBackup

grid Grid-peer gridse001

wn001

wn002

wn024

wn025

Switch-1

GRID-PEER Tier-3 cluster with 32 & 64 bit machine

Computing Nodes

IN-DAE-VECC-02 Site with 64 bit machine

Installer

DELL and HP Blade Server with Multi Core Xeon 3.0 GHz

naamak Dache-server grid01

10G Backbone Switch

4 – Xrootd Disk Servers

SINP1Gbps Fiber Backbone

10 G

bps

10 G

bps

10G Backbone Switch



From 128Kbps to 1Gbps WAN Network

Started with

----128Kbps shared link 2002

----512Kbps 2003

----2Mbps Dedicated Link 2004

----4Mbps from Bharti 2006

----30Mbps from Reliance 2008

----100Mbps from VSNL (ERNET) 2009

----300 Mbps from NKN 2011

----1Gpbs 2014



# cat route-eth0default via 192.168.1.12 dev eth0144.16.112.16 via 192.168.1.16 dev eth0144.16.112.15 via 192.168.1.15 dev eth0144.16.112.14 via 192.168.1.14 dev eth0144.16.112.9 via 192.168.1.9 dev eth0

Kolkata SE Traffic


Tier-3 (grid-peer) cluster

1U Sliding LCD Monitor with 16 port KVMDell PowerEdge M1000e Blade Server Chassis.16 Number of Dell PowerEdge M610 High Performance Intel BladeEach blade has Nehalem based 2 * Intel Quad Core E5530 Xeon 2.4GHz CPU with 8MB cache.Each blade has16GB RAM.Each blade has 2 * 146GB Mounted as RAID1.Installed SLC 6.5 x86_64 OS

IBM SAN Storage3 * 16 * 2TB SAS hard disks.3 * 26 TB = 78 TB Usable space after RAID5 and Hot Spare.

Total 14 Nodes for VECC users and Collaborating Institutes.

13 Collaborating Institutes.

64bit computing nodes.

7 Non-interactive AMD nodes.

7 Interactive AMD + Intel nodes.

78 TB IBM SAN storage.

Storage mounted via NAS on all the Compute Nodes.

10Gbps Network backbone.

Managing via ROCKS6.

Using PBS Job Scheduler.

50 + active users (across India.)

30 + active users (in VECC.)

Quota implemented for the users.

Root, Geant3, Aliroot, Alien, Fairsoft, CBMroot, gFortran etc user specific software installed.

HP c7000e Blade enclosure.8 Number of HP ProLiant DL585 G7 AMD BladeEach blade has 2 * 16 Core 2.5GHz AMD CPUEach blade has 256 GB RAM.Installed SLC 6.5 x86_64 OS


Decided to increase Kolkata Tier-2 resources as per ALICE

Fair Share rule.

10 Cooled Rack Space allotted.

Indent raised to procure more than 30K HEP Spec 2006

equivalent computing Power.

Will procure 1200 Physical cores or 2400 Hyperthreaded core.

12 Core processors are the best suitable as per price

performance point.

Plan to purchase only bare disks as per requirement.

3 * 2U servers comprising 12*4TB disks configured under

Open Source EOS storage, will be included into KOLKATA

SE.

Future Plan and Decisions



Network Backbone Planning

10Gb backbone,40Gb core switch connection.Both Fiber and Copper connectivity.Connection via DAC Cable.Dual path, Full Redundancy.



Thank YouDURGA POOJAFestival of GODESS DURGA.Celebrated in Month of October.Idol will be submerged into the Water.

kolkata tier-2@alice grid site name :- in-dae-vecc-02monalisa name:- kolkata-cream vo :- alicecity:-...

Documents