Provision of networks for the LHC experiments in Germany
- Status 03/2006 -
K. Schauerhammer, K. Ullmann, (DFN)
2. / 3.3.2006
Meeting GridKa TAB
Karlsruhe
Seite 2
Contents
I. Technical options X-WiN and Geant2
II. Architecture LCG - Network (T0 - T1)
III. Performance Measurements
DFNInternet
IV. Connecting T2 centres in Germany
Seite 3
I. Technical options X-WiN and Geant2
Seite 4
X-WiN - status as of 1. 1. 2006
• Technical concept backbone (after Europe wide tender)– backbone consists of (dark) fibre and leased circuits– operational responsibility: DFN– more (than before) bought in (partial) services– much more (than before) DFN PoPs
• Access-Network very similar as in the past• Economic concept
– backbone put together from different service offerings from the market and integrate that under DFN responsibility
– most effective usage of competition on the market
Seite 520.02.2006
X-WiN (optical platform)
GAR
ERL
BAY
MUE
FZJ
AAC BIR
DES
HAM
POTTUB
FZK
GSI
DUI
BRE
HAN
BRA MAGBIE
FRA
HEI
STU
REG
DRE
CHE
ZIB
ILM
KIE
ROS
LEI
JENWEI
ESF
HUB
ADH
KAI
GOEKAS
MAR
GIE
Dark fibre
Leased circuits
GRE
80km toFrankfurt/O(Pionier/PL)
120km to Zürich(Switch/CH, GARR/IT)
50km toKehl/Straßburg
(Renater/FR)
50km toEnschede
(Surfnet/NL)
43 DFN PoPs
Seite 6
Optical platform based services
• Toolbox for the provision of– DFNInternet (2Mbps to 10 GE)– VPN-Services based on optical links– services like DFNVC, DFNPKI, DFN-CERT,
DFNRoaming, DFNNews etc. unchanged
• New cost structures for optical networks enable economic solutions for specialised services (Optical Private Networks / OPNs) i.e. Grids
Seite 7
General targets X-WiN design
• More performanceperformance increase by a factor of 4 since 01/06
more performance available (for the same price) as of 01/07
• More flexibilityno volume charging (and limitation)
Ethernet as additional access technology
Hybrid PoPs enable VPNs
• More availibilityduring design for backbone implicitly taken into
account
Seite 8
Availibility backbone by means of ...
GAR
ERL
BAY
MUE
FZJ
AAC BIR
DES
HAM
POTTUB
FZK
GSI
DUI
BRE
HAN
BRA MAGBIE
FRA
HEI
STU
REG
DRE
CHE
ZIB
ILM
KIE
ROS
LEI
JEN
ESF
HUB
ADH
• strongly protected physical topology – protected fibre paths– meshed fibre ring structure– more mathematical
optimisation will show, how to meaningful complement physical topology
Seite 9
Availibility backbone by means of...
• Fault tolerant logical topology, especially for DFNInternet
• Well-known optimisation for – logical links between PoPs – best mapping to physical topology
• Optimisation targets, especially– minimal delay– minimal loss of connectivity in case of severe
failures– best („smooth“) utilisation in case of failures
Seite 10
X-WiN DFNInternet as of 01/06
Logical IP backbone
Seite 11
worldwide Internet
worldwide Internet
Availibility backbone by means of...
User
X-WiN
At least two accesses to global IP networks, more redundant carriersFull throughput in case of failuresAdditional roughly 100 direct peerings
Redundant global connectivity by means of independant paths and PoPs assures protection against problems with Carriers (e.g. network failures or insolvency)
Redundant global connectivity by means of independant paths and PoPs assures protection against problems with Carriers (e.g. network failures or insolvency)
Redundant global connectivity
Seite 12
Dark Fibre
• Technical definition– ITU-T conform and "WDM-applicable"– some minimal requirements on technical
parameters• for example attenuation
• Service: – Support : 24/7 hotline and debug– provision of colocation space – End-to-end operational responsibility– usual time for reaction– integration into DFN defined processes
Seite 13
Equipment to „light“ Dark Fibre
• Technical definition:– equip with (several optional) digital interfaces– provision of a management system
• Service:– includes installation and maintenance – support: 24/7 hotline and debug– usual time for reaction– integration into DFN defined processes
Seite 14
Operations
• Definition– Monitoring of management data– pursue well defined procedures
• failure identification (devices? - links?)• alarm of responsible support groups• supervision of debug• if necessary escalation
– all procedures DFN defined– regular (i.e. monthly) reporting– open for integration of new devices
Seite 15
PoP A
PoP B
PoP C
Operational model X-WiN platform
dark fiber
Leased circuit
Access to management dataof DFN „owned“ devices
Operations Groupalarms and triggers support(links, devices)
Seite 16
GÉANT2 footprint
LU
RU
EELV
LT
DK
IT
FR
BE
CH
SI
AT
HR
PLDE
CZ
PT ES
IENL
UK
RO
BG
TR
CYIL
MT
GR
HU
SK
2
2
3
3
3
3
2
2
22
2
3
3 2
2
2
222
1 1
11
1
1
Seite 17
GÉANT2 Hybrid POP
Dark Fibre
Dark Fibre
N x 10G N x 10G
10G N x 10G
GÉANT2 POP
to NREN
Leased Service
Seite 18
Gigabit Ethernet p2p Service
GÉANTBorder
GÉANTBorder
Interconnect:N x physical GEth
GÉANT2
Interconnect:N x 10GEth
VLAN/10GEth(LAN or WAN)
Physical GEth
GEth (NREN transport)
GEth (GÉANT2 transport)
NREN A NREN B
Seite 19
II. Architecture LCG - Network (T0 - T1)
Seite 20
Architecture LCG network (1)
• Assumptions, design criteria:– bandwidths 10Gbit/s per T1 resp. multi 10Gbit/s
for T0– continuous stream of data– „keep it simple“ (as possible)– organise transport if possible in layer 2– security already considered during design
• system should be protected from unauthorised access • access from “trusted sources” not limited
Seite 21
Architecture LCG Network (2)
• Optical Private Network, consists of dedicated 10G links from T0 and any T1,
• T1 access in two options:– “Light path T1”– “Routed T1”
• Back-up for T0-T1 is part of the design
• T0 interface very likely 10GE LAN-PHY
Seite 22
Scalability of the architecture
• Options for transfer:– 10 Gbit/s ~ 1014 byte/d or 100 Tbyte/d– eleven 10G links -> more than 1 Pbyte/d or
roughly 0.5 Exabyte/a
• If one 10G link per T1 is not sufficient:– Installation of just another 10G link T0 - T1 if
possible on physically separated fibre path– architecture covers this completely
Seite 23
LCG T0 - T1 OPN
Seite 24
III. Performance Measurements DFNInternet
Seite 25
Issues in setting up a 10Gbit connetion between GridKa
and CERN openlab Forschungszentrum Karlsruhe GmbH
Institute for Scientific ComputingP.O. Box 3640
D-76021 Karlsruhe, Germanyhttp://www.gridka.de
Bruno Hoeft
Taken from:
Measurements DFNInternet
Seite 26
CERNGridFTPserver
GridFTPserver
WAN 2003/4 -- Gigabit GridKa – CERN
Géant10 Gbps
DFN
2.4 Gbps
GridFTP tested over 1 Gbps
Karlsruhe
Frankfurt
2x 1 Gbps
98% of 1Gbit
Geneva
1000
0
Seite 27
10Gigabit WAN SC GridKa – CERN
CERNGridFTPserver
GridFTPserver
Géant10 Gbps
DFN
10 Gbps
10 Gbps 10 Gbps
Karlsruhe
Frankfurt
- Bandwidth evaluation (tcp/udp)- MPLS via France (MPLS - MultiProtocol Label Switching)
- LBE (Least Best Effort)
- GridFTP server pool HD to HD Storage to Storage - SRM
Geneva
Seite 28
Evaluation of max. throughput
700
1400
5600
4900
4200
3500
2800
2100
7000
6300
Mbit/s
18:00 20:00
9 nodes each site-8 * 845 Mbit-1 * 540 Mbithigher speed at one stream is resulting in a packet loss
Seite 29
IV. Connecting T2 centres in Germany
Seite 30
LCG network services in Germany
• T0-T1 problem– 10 Gbit/s over European VPN (GridKa - CERN)– backup via CNAF or SARA
• T1-T2 problem– within DE no problem as most LHC sites are
close to X-WiN PoPs– for T2 in DE rough guess of traffic flows are
needed (source, destination, transferred volume per day, peak requirements?)
Seite 31
DESHAM
ZIB
HAN
LEIDUI
BIR
FRA
ERL
GAR
STU
MUE BIE
ROS
DORKAS
KOE
AAC
BON
FZK
TUM
AUG
REG
BAM
CHE
DREBOC
WUP
HEI
GSI
MAI
FRE
MAN
SIE
BRE
X-WiN-Nodes
LHC User Locations at X-WiN-Nodes
LHC User locations to be connected to X-WiN nodes
LCG Sites DE and X-WiN