![Page 1: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/1.jpg)
Evolution of the Italian Tier1 (INFN-T1)Evolution of the Italian Tier1 (INFN-T1)
Umea, May [email protected]
1
![Page 2: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/2.jpg)
May 2009 2
In 2001 the project of the Italian Tier1 in Bologna at CNAF was born.
First computers were based on Intel Pentium III @ 800 MHz, 512 Mbytes of RAM , 20 GBytes of local storage and 100 Mbit/sec network interface.
No redundancy: 1 chiller, 1 electric line power, 1 static UPS (based on chemical batteries), 1 motor generator diesel engine, no remote control.
Maximum usable electric power was 1200 KVA, that means maximum 400 KVA for computers.
Situation not compatible with HEP experiment requests.
We decided to rebuild everything!
In 2008 works started for the new mainframe rooms and new power station.
Today all works are finished, all is up&running.
We are on time and the original schedule was fully respected.
Total infrastructure redundancy and total remote control.
2
![Page 3: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/3.jpg)
May 2009 3 3
New infrastructure is based on 4 rooms:
1)Chiller cluster (5 + 1)
2)2 Mainframe rooms + Power Panel Room
3)1 Power station
4)2 Rotary UPS and 1 diesel motor generator
Chiller cluster: 350 KWatt cooled per chiller: in best effiency (12C in 7C out) ratio 2:1, 50 KWatt needed to cool 100 KWatt
Mainframe: more than 100 (48 + 70) rack APC Netshelter SX 42U 600mm Wide x 1070mm deep Enclosure.
15 KVolt Power station 3 Transformer (2 + 1 redundancy, 2500 KVA each one) Rotary UPS (2 x 1700 KVA) 2 electrical lines (red and green, 230 Volt per phase, 3 phases + ground).
Motor generator diesel engine: 1250 KVA (60.000 cc)
3
![Page 4: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/4.jpg)
Sala chiller (piano -1)
CHI LLER
CHI LLER
CHI LLER
CHI LLER
CHI LLER
CHI LLER
CHI LLER
CHI LLER
CHI LLER
CHI LLER
Quadri elettrici dei chiller
Sala chiller (piano -1)Sala calcolo (piano -2)Spazi consegna elettr. Spazi emergenza elettr.
4
Quadrielettrici
Tubi H2O
Canaliuscita aria
Muro
Chiller
Settifonoass.
Torreuscita aria
Flussouscita aria
![Page 5: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/5.jpg)
Sala calcolo (piano -2)Sala chiller (piano -1)Sala calcolo (piano -2)Spazi consegna elettr. Spazi emergenza elettr.
5
Blindo bar
Quadri (power panel)
Rack
![Page 6: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/6.jpg)
Spazi consegna elettr.
V i a R a n z a n i
ENEL
15KV
Cabina consegna Corridoio cavi 15KV
Sala ditrasformazione
TR2
TR1
TR3
Quadri elettrici
Quadri
elettrici
Sala chiller (piano -1)Sala calcolo (piano -2)Spazi consegna elettr. Spazi emergenza elettr.
6
Collaudo elettrico
![Page 7: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/7.jpg)
Spazi emergenza elettr.
2 gruppi rotanti
(rotary UPS)
2 cisterne gasolio
(diesel oil tank)
Sala chiller (piano -1)Sala calcolo (piano -2)Spazi consegna elettr. Spazi emergenza elettr.
7
![Page 8: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/8.jpg)
8
Sala chiller (piano -1)Sala calcolo (piano -2)
Spazi consegna elettr.
Spazi emergenza elettr.
![Page 9: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/9.jpg)
9
Sala chiller (piano -1)Sala calcolo (piano -2)
Spazi consegna elettr.
Spazi emergenza elettr.
![Page 10: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/10.jpg)
Foto impianto H2O
10
![Page 11: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/11.jpg)
Foto chiller
![Page 12: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/12.jpg)
Foto rack
![Page 13: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/13.jpg)
13
Cooling and AC test
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
A/C
4kWh4kWh
4kWh4kWh
4kWh4kWh
4kWh4kWh
4kWh4kWh
4kWh4kWh
4kWh4kWh
4kWh4kWh
4kWh4kWh
4kWh4kWh
![Page 14: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/14.jpg)
Remote Control
May 2009 14
![Page 15: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/15.jpg)
Remote Control
May 2009 15
![Page 16: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/16.jpg)
Remote Control
May 2009 16
![Page 17: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/17.jpg)
Network Upgrade: New Core Switch
17
Buy of the new Core Switch is in progress.
Minimum request:
• 64 fiber ports 10 Gbit/sec (over subscriptions allowed 4:1)
• 90 fiber ports 1 Gbit/sec
Allowed upgrade requested:
• 100 Gbit/sec fiber ports
• 128 fiber ports 10 Gbit/sec without over subscriptions
![Page 18: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/18.jpg)
INFN-T1 WAN connection today
• INFN-T1 CERN (T0): 10 Gbit/sec
• INFN-T1 FZK (T1): 10 Gbit/sec [backup of INFN-T1 CERN] + IN2P3 + SARA link
• INFN-T1 General purpose: 10 Gbit/sec
May 2009 18
![Page 19: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/19.jpg)
Farming
May 2009 19
![Page 20: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/20.jpg)
KSI2K used in 2009
May 2009 20
![Page 21: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/21.jpg)
KSI2K used by VOs
May2009 21
![Page 22: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/22.jpg)
Farming upgrade
• Q4/2009: +20.000 HEP-SPEC installed• No more FSB technology (Nehalem and Shanghai only)
• Option to buy in 2010 directly (less bureaucracy)
• Remote Console
• ~50% for LHC experiments
• We support more than 20 experiments/collaborations
May 2009 22
![Page 23: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/23.jpg)
Farming: HEP-SPEC
May 2009 23
![Page 24: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/24.jpg)
Disk Storage Systems
May 2009 24 24
All systems interconnected in a SAN 12 FC switches (2 core switches) with 2/4 gbps connections ~ 200 disk-servers ~ 2.6 PB raw (~ 2.1 PB-N) of disk-spac
13 EMC/DELL Clarion CX3-80 systems (SATA disks) interconnected to SAN 1 dedicated to databases (FC disks) ~ 0.8 GB/s bandwidth 150-200 TB each 12 disk-servers (2 x 1 gbits uplinks + 2 FC4 connections) part configured as gridftp
servers (if needed), 64 bits OS (see next?? slide)
1 CX4-960 (SATA disks) ~ 2 GB/s bandwidth 600 TB
Other older hw (Flexline, Fast-T etc..) being progressively phased out
No support (part used as cold spares) Not suitable for GPFS
![Page 25: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/25.jpg)
Tape libraries
May 2009 25 25
New SUN SL8500 in production since July’08 10000 slots 20 T10KB drives (1 TB) in place
8 T10KA (0.5 TB) drives still in place 4000 tapes on line (4 PB)
“old” SUN SL5500 1 PB on line, 9 9940b and 5 LTO drives Nearly full No more used for writing
Repack on going ~5k 200 GB tapes to be repacked ~ 500 GB tapes
![Page 26: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/26.jpg)
How experiments use the storage
• Experiments present at CNAF make different use of (storage) resources – Some use almost only the disk storage (e.g. CDF, BABAR)– Some use also the tape system as an archive for older data (e.g. VIRGO)– The LHC experiments exploit the functionalities of the HSM system…– ….but in different ways
• CMS and Alice) use primarily the disk with tape back-end, while others Atlas and LHCb concentrate their activity on the disk-only storage (see next slide for details)
• Standardization over few storage systems, protocols– Srm vs. direct access– file, rfio, as LAN protocols– Gridftp as WAN protocol– Some other protocols used but not supported (xrootd, bbftp)
May 2009 26
![Page 27: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/27.jpg)
STORAGE CLASSES
• 3 class of services/quality (aka Storage Classes) defined in WLCG • Present implementation at CNAF of the 3 SC’s
– Disk1 Tape0 (D1T0 or online-replica) GPFS/StoRM• Space managed by VO • Mainly LHCb, Atlas, some usage from CMS and Alice
– Disk1 Tape1 (D1T1 or online-custodial) GPFS/TSM/StoRM• Space managed by VO (i.e. if disk is full, copy fails)• Large buffer of disk with tape back end and no garbage collector• LHCb only
– Disk0 Tape1 (D0T1 or nearline-replica) CASTOR• Space managed by system• Data migrated to tapes and deleted from disk when staging area full • CMS, LHCb, Atlas, Alice • testing GPFS/TSM/StoRM
This setup satisfies nearly all WLCG requirements (so far) exception made for:• Multiple copies in different Storage Area for a sURL• Name space orthogonality
May 2009 27
![Page 28: Evolution of the Italian Tier1 (INFN-T1) Umea, May 2009 Felice.Rosso@cnaf.infn.it 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551c046d550346b24f8b4dcf/html5/thumbnails/28.jpg)
Questions?
May 2009 28