status and plans

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Status and Plans

TERENA 2010Vilnius, Lithuania

John Shade /CERN

CERN IT Department

CH-1211 Genève 23


it

Large Hadron Collider

LHCOPN Presentation, TERENA 2010 – Slide 2

CERN IT Department

CH-1211 Genève 23


it

CERN Accelerator Complex


CERN IT Department

CH-1211 Genève 23


it

26659m in Circumference

5000 SC Magnets pre‑cooled to -193.2°C (80 K)using 10 080 tonnes of liquid nitrogen

60 tonnes of liquid helium bring them down to -271.3°C (1.9 K).

In 1 second, a proton will circulate the LHC 11245 times

The internal pressure of the LHC is 10-13 atm, ten times less than the pressure on the Moon

600 Million Proton Collisions/second

LHC (Some) Facts and Figures


CERN IT Department

CH-1211 Genève 23


it

CERN’s LHC Detectors


ALICE

26 m long 16 m high 16 m wide10 000 tonnes

CMS

21 m long 15 m high 15 m wide12 500 tonnes

LHCb

21m long 10m high 13m wide5600 tonnes

46 m long 25 m high 25 m wide7000 tonnes

ATLAS

CERN IT Department

CH-1211 Genève 23


it

30-MAR-2010 – collisions at 7TeV


On 30 March 2010, beams collided in the LHC at 7 TeV, the highest energy ever achieved in a particle accelerator, marking a new world record and the start of the LHC research programme.

More than half a billion collisions observed to date. Physicists from all over the world are analysing the new data and retracing the particles discovered in past experiments (e.g. W particle and the B-meson).

CERN IT Department

CH-1211 Genève 23


it

ALICE Events


CERN IT Department

CH-1211 Genève 23


it

ATLAS Collisions


CERN IT Department

CH-1211 Genève 23


it

CMS Collisions


CERN IT Department

CH-1211 Genève 23


it

LHCb Events


CERN IT Department

CH-1211 Genève 23


it

Detectors ~150 million electronic channels40 MHz collision rate

Fast response electronics,FPGA, embedded processors,very close to the detector

N x 10 Gbit links to the Computer Center

LHC

Accelerator and 4 Experiments

Level 1 Filter and Selection

Level 2 Filter and Selection

1 Pbytes/s

150 Gbytes/s

0.7 Gbytes/s

CERN Computer Center

Event selection is based on the physics model,(“prejudice”, expectations)Will change over time,“limits” the physics


Raw Data Rates

CERN IT Department

CH-1211 Genève 23


it

4 Experiments

3 PBytes/s

We are looking for 1 ‘good’ snapshot in 10 000 000 000 000 ‘photos’

2 GBytes/sto the CERN computer center

Store on disk and tape

World-Wide Analysis

Export copies

Create sub-samples

col2f

2f

3Z

ff2Z

ffee2Z

0

ff

2z

2Z

222Z

2Z0

ffff

N)av(26

m and m12

withm/)m-(

_

__

FG

sss

PhysicsExplanation of nature

10 GBytes/s 4 GBytes/s

1 TByte/s ?Distributed + local

The Dataflow

Filter and first selection

The Data Flow


CERN IT Department

CH-1211 Genève 23


it

WLCG Project

• The Worldwide LHC Computing Grid Project, WLCG, (http://lcg.web.cern.ch/LCG/) is a global collaboration of more than 140 computing centres in 34 countries.

• The mission of the WLCG project is to build and maintain a data storage and analysis infrastructure for the entire high energy physics community that will use the Large Hadron Collider at CERN.


CERN IT Department

CH-1211 Genève 23


it

Tier0 center, CERN very large center; tape storage; 1st-level processing and meta-data

storage; quality service (24*7)

• Tier1 center, 11 world-wide large capacity; tape storage; quality service

• Tier2 center, 129 world-wide medium size, some large; no 24*7 service; no custodial storage

• Tier3 center very small to medium size; no availability guarantee; focus on end-user analysis activity

Tier Structure


CERN IT Department

CH-1211 Genève 23


it

The Beginning ...

• Essential for Grid functioning to distribute data out to the T1’s.– Capacity had to be large enough to deal with most situations

including “Catch up”• LHCOPN proposed in 2004 by D. Foster (CERN) as a

“Community Network”– Renamed as “Optical Private Network” as a more descriptive name

after the initial meeting of stakeholders in Amsterdam.– Based on 10G as the best choice for affordable adequate

connectivity by 2008.• Considered by some as too conservative - can fill a 10G pipe with just (a few)

PC’s!• 10G is commodity now!

• Simple end-end model– This was not a research project, but, an evolving production network

relying on emerging facilities.


CERN IT Department

CH-1211 Genève 23


it

Issues, Risks, Mitigation

• It is a complex multi-domain network relying on infrastructure provided by:– (links) NREN’s, Dante and commercial providers– (L3) T1’s and CERN– (operations) T1’s, CERN, EGI and USLHCNet

• Managed by the community– “Closed Club” of participants– Simple L3 model, routers at the end points – Federated operational model

• Design separated from implementation• Need to combine innovation and operation


CERN IT Department

CH-1211 Genève 23


it

LHCOPN L2 Network Map


CERN IT Department

CH-1211 Genève 23


it

Current Situation

• T0-T1 Network is operational and stable.• Several areas of focus:

– Physical Path Routing– Operational Support– Monitoring


CERN IT Department

CH-1211 Genève 23


it

Operational Model

• GGUS used as ticket & tracking system• Twiki used for collaboration documents

• Incident Management procedures• Change Management procedures

• Conference calls used for Operations and Monitoring, with F2F meetings on a regular basis


CERN IT Department

CH-1211 Genève 23


it

T0-T1 links’ availability


ExplanationLink availability calculated on the uptime of the BGP routing protocol relationship with each neighbor. It says for how long the T0's routers were able to route traffic to/from every T1 over the LHCOPN link. It doesn't say anything about Data-Centre to Data-Centre connectivity.

FNAL-Sec and BNL-sec are the additional 10G links recently deployed, thus the high “Time Undertermined”.

CERN IT Department

CH-1211 Genève 23


it

Basic Link Layer Monitoring

• perfSONAR - Integrated into the “End to End Coordination Unit” (E2ECU) run by DANTE– Provides simple indications of “hard” faults.– Insufficient to understand the quality of the

service• Dante MDM deployed (but is it useful?)

– Configuration audit underway. • Specifications for visualisation dashboard(s)

being developed– Access to the MDM data is needed


CERN IT Department

CH-1211 Genève 23


it

LHCOPN Traffic

LHCOPN Presentation, TERENA 2010 – Slide 22http://network-statistics.web.cern.ch/network-statistics/ext/?p=sc

CERN IT Department

CH-1211 Genève 23


it

LHCOPN Core Weekly Traffic


CERN IT Department

CH-1211 Genève 23


it

LHCOPN Yearly Traffic Volume


CA-TRIUMF

DE-KIT

ES-PIC

FR-IN2P3

IT-INFN-CNAF

NDGF

NL-T1

TW-ASGC

UK-T1-RAL

US-FNAL-CMS

US-T1-BNL

0 1000 2000 3000 4000 5000 6000

Volume From T1s (TB)Volume To Tier1s (TB)

Tera Bytes

Traffic exchanged among the T0 and the Tier1s and transited through the CERN routers. Period: 05-2009 to 05-2010(source: http://network-statistics.web.cern.ch/network-statistics/ext/?p=sc&q=LHCOPN%20Total%20Traffic&m=LHCOPN-Total)

CERN IT Department

CH-1211 Genève 23


it

Link capacity


• LHCOPN uses 10Gb/s links between Tier0 and all Tier1s• For BNL, FERMI and RAL, two 10Gbps

links are configured in round-robin, thus providing 20Gbps

• Renater dark fibre to Lyon would allow IN2P3 to upgrade

• We await (next year?) to see the prices of 40Gb interfaces. Will they be cheaper than 4*10?

CERN IT Department

CH-1211 Genève 23


it

Future Plans


• LHCOPN “core” (Tier0, Tier1s):• Extend capacity for Tier1-Tier1?• Extended deployment of Cross-Border

Fibre?• Dedicated Tier1 Exchange Point (T1XP)?

• Use LHCOPN operational model for Tier2s?• Hard to impose a central model on Tier2s

CERN IT Department

CH-1211 Genève 23


it

Conclusion


• Current LHCOPN (core) does what it was designed for - shouldn’t be modified (extend – not redesign)

• We should care about the “broader picture”- data distribution and movement does not

stop at Tier1s and is important for LHC experiments’ operation

• Tier1-Tier1 and Tier1-Tier2 data movements will need to be addressed, probably sooner rather than later

CERN IT Department

CH-1211 Genève 23


it

Conclusions (continued)


• Experiments’ use of the OPN is exceeding initial estimates

• We need to foresee growth of connectivity to 40G and 100G and explore the paradigms of dynamic circuit provisioning • Transatlantic connectivity will be a challenge!

status and plans

Documents

lithuania john shade

cern computer centerstore

figureslhcopn presentation

physicslhcopn presentation

tevlhcopn presentation

tonnes cms21

tonnes lhcb21m

wide5600 tonnes