us lhcnet update dan nae california institute of technology dan.nae@cern.ch lhc opn meeting munich,...
Post on 29-Jan-2016
217 Views
Preview:
TRANSCRIPT
US LHCNet UpdateUS LHCNet Update
Dan NaeDan Nae
California Institute of TechnologyCalifornia Institute of Technology
dan.nae@cern.chdan.nae@cern.ch
LHC OPN MeetingLHC OPN MeetingMunich, April 2007Munich, April 2007
LHC OPN Meeting, Munich
What is the US LHCNetWhat is the US LHCNet
A transatlantic network designed to support A transatlantic network designed to support the the LHCLHC and the U.S. HEP community and the U.S. HEP community
Funded by the US Funded by the US DoEDoE and and CERNCERN and and managed by managed by CaltechCaltech in collaboration with in collaboration with CERNCERN
Evolved from a network between US and Evolved from a network between US and CERN which dates back to 1985CERN which dates back to 1985
Our mission is to deliver a reliable network Our mission is to deliver a reliable network service to support the upcoming LHC service to support the upcoming LHC experiments at CERNexperiments at CERN
Designed to support the LHC three-tiered Designed to support the LHC three-tiered model and to deliver data directly to/from the model and to deliver data directly to/from the US Tier 1’sUS Tier 1’s ( (FNALFNAL and and BNLBNL))
LHC OPN Meeting, Munich
Atlantic Ocean
Circuit StatusCircuit Status
NYC 111 8th
Pottington (UK)
VSNL South
NY60 Hudson
HighbridgeVSNL North
AMS-SARAAC-2
Bude
GVA-CERN
FrankfurtVSNL
Wal, NJ
London
Global Crossing
Qwest
Colt
GEANT
NYC-MANLAN
CHI-Starlight
Paris
BellportWhitesands
Unprotected circuits (lower cost) Service availability from provider’s offers:
Colt Target Service Availability is 99.5% Global Crossing guarantees Wave Availability at 98%
Canarie and GEANT: No Service Level Agreement (SLA)
LCG Availability LCG Availability requirement: 99.95%requirement: 99.95%
LHC OPN Meeting, Munich
Circuit Status (cont.)Circuit Status (cont.)
- 6 x 10 Gbps6 x 10 Gbps SONET/10GbE WAN-PHY. SONET/10GbE WAN-PHY.- AmsterdamAmsterdam PoP installed in January. PoP installed in January.- Last circuit (Last circuit (Geneva-AmsterdamGeneva-Amsterdam) )
delivered at the end of last month.delivered at the end of last month.- Not as diverseNot as diverse as one might expect. as one might expect.- We’ve never had an isolated node so far.We’ve never had an isolated node so far.- It happened more than once that the It happened more than once that the
circuits went down in pairs (GC NYC-CHI circuits went down in pairs (GC NYC-CHI and NYC-AMS, Qwest NYC-CHI and CHI-and NYC-AMS, Qwest NYC-CHI and CHI-GVA).GVA).
- Rumors of a new RFP in May-June?Rumors of a new RFP in May-June?
LHC OPN Meeting, Munich
Circuit Availability (March)Circuit Availability (March)
Proposal to produce a monthly reportProposal to produce a monthly report
LHC OPN Meeting, Munich
““Old” Network Map (March ’07)Old” Network Map (March ’07)
LHC OPN Meeting, Munich
Equipment StatusEquipment Status
- Next generation network based on Ciena Next generation network based on Ciena CoreDirector/CI – CoreDirector/CI – VCAT/LCAS/GFP capabilities.VCAT/LCAS/GFP capabilities.
- Four nodesFour nodes, one at each PoP to complement the , one at each PoP to complement the existing existing Force10Force10 boxes. boxes.
- Number of links on each box expected to Number of links on each box expected to increase each year; the upgrade plan has been increase each year; the upgrade plan has been discussed with Ciena up to discussed with Ciena up to ~2010.~2010.
- Deployment has already started (the Deployment has already started (the New YorkNew York and and ChicagoChicago nodes were installed, tested and nodes were installed, tested and commissioned by the commissioned by the end of last monthend of last month).).
- The The GenevaGeneva node is physically installed, will be node is physically installed, will be tested and most likely accepted next week.tested and most likely accepted next week.
- The fourth node (The fourth node (AmsterdamAmsterdam) expected to be ) expected to be operational in May.operational in May.
LHC OPN Meeting, Munich
Past and Present IssuesPast and Present Issues
- At least one At least one Stratum 1 PRSStratum 1 PRS is recommended for normal is recommended for normal operation (still to be installed)operation (still to be installed)
- Second Second DC power feedDC power feed in Chicago is still to be installed in Chicago is still to be installed (today in non-redundant mode)(today in non-redundant mode)
- One ESLM card (10GbE) locked up during the migration of One ESLM card (10GbE) locked up during the migration of the the NYC-CHI GC circuit (fixed by hardware reset)the the NYC-CHI GC circuit (fixed by hardware reset)
- DC rectifier at CERN not fully populated (but enough for DC rectifier at CERN not fully populated (but enough for the moment)the moment)
- CERN commissioning has been CERN commissioning has been delayeddelayed due to due to complicated site access procedurescomplicated site access procedures
- We need more We need more operational experienceoperational experience with the boxes with the boxes- Currently only one circuit (GC NYC-CHI) runs over the new Currently only one circuit (GC NYC-CHI) runs over the new
equipment (migrated on equipment (migrated on March 29March 29thth))
LHC OPN Meeting, Munich
Equipment Status TodayEquipment Status Today
- Three nodes delivered (GVA, CHI, NYC)Three nodes delivered (GVA, CHI, NYC)
- Two nodes installed (NYC, CHI)Two nodes installed (NYC, CHI)
- One circuit migrated (GC NYC-CHI)One circuit migrated (GC NYC-CHI)
LHC OPN Meeting, Munich
Proposed Migration (May)Proposed Migration (May)
Two complementing networks:
- first network is essentially last year’s US LHCNet – stable, reliable, dedicated 10Gbps T0-T1 (and T1-T1)
- the second is based on the new equipment – dynamic or static point-to-point VCs between any of the four nodes
- the “new network” can be reconfigured to carry T0-T1 traffic in case of problems
Equipment diversity (more robust)
LHC OPN Meeting, Munich
GEANT-ESNET Peering over US GEANT-ESNET Peering over US LHCNetLHCNet
The idea came up last year during the meeting at The idea came up last year during the meeting at FNALFNAL
US LHCNet would transparently carry a L1/L2 US LHCNet would transparently carry a L1/L2 Virtual CircuitVirtual Circuit between NYC and Amsterdam between NYC and Amsterdam
GEANT and ESNET peer over this VCGEANT and ESNET peer over this VC The The bandwidthbandwidth of the circuit is still to be of the circuit is still to be
discusseddiscussed An An MOUMOU between between ESNETESNET and and US LHCNetUS LHCNet was was
signed (with DOE’s blessing)signed (with DOE’s blessing) ESNETESNET will install a new 10 GbE connection to will install a new 10 GbE connection to
the US LHCNet CD/CI in the US LHCNet CD/CI in NYCNYC for this purpose for this purpose A new 10 GbE connection in A new 10 GbE connection in AmsterdamAmsterdam
between US LHCNet and between US LHCNet and GEANTGEANT is required is required
LHC OPN Meeting, Munich
Proposed Migration (cont.)Proposed Migration (cont.)
Additional links are needed to make full use of the new infrastructure: Additional links are needed to make full use of the new infrastructure:
-New BNL/ESNET link in NYCNew BNL/ESNET link in NYC
- New FNAL/ESNET link in Chicago (or existing to be moved to the CD/CI)New FNAL/ESNET link in Chicago (or existing to be moved to the CD/CI)
- New LCG – CD/CI link(s) in GenevaNew LCG – CD/CI link(s) in Geneva
- New GEANT & SARA links in Amsterdam (T1-T1 and peerings)New GEANT & SARA links in Amsterdam (T1-T1 and peerings)
LHC OPN Meeting, Munich
Network Forecast - 2007Network Forecast - 2007
Geneva
Chicago
New York
4 x 10GBE4 x 10GBE
4 x 10GBE
1 x 10G SDH
2 x 10G SDH
1 x 10G SDH
ESnet SDN
Amsterdam
1 x 10G SDH
2 x 10GBE
1 x 10G SDH
1 x 10G SDH 1 x 10G SDH
Ports Ports at Each at Each
PoPPoP
10 10 GbEGbE
OC-192OC-192/STM-64/STM-64
AMSAMS 22 22
GVAGVA 44 33
NYCNYC 44 55
CHICHI 44 44
Mostly sunny, scattered Mostly sunny, scattered cloudsclouds
LHC OPN Meeting, Munich
Network Forecast - 2008Network Forecast - 2008
Geneva
Chicago
New York
4 x 10GBE
4 x 10GBE
6 x 10GBE
2 x 10G SDH
1 x 10G SDH
ESnet SDN
1 x 10G SDH 1 x 10G SDH
2 x 10G SDH Amsterdam
1 x 10G SDH
2 x 10GBE
2 x 10G SDH
1 x 10G SDH 1 x 10G SDH
I2 NewNet
Ports at Ports at Each PoPEach PoP
10 GbE10 GbE OC-192OC-192/STM-64/STM-64
AMSAMS 22 33
GVAGVA 66 55
NYCNYC 44 77
CHICHI 44 55
Clouds gathering…Clouds gathering…
LHC OPN Meeting, Munich
Network Forecast - 2009Network Forecast - 2009
Geneva
Chicago
New York
4 x 10GBE
4 x 10GBE
8 x 10GBE
3 x 10G SDH
3 x 10G SDH
2 x 10G SDH
ESnet SDN
1 x 10G SDH 1 x 10G SDH
2 x 10G SDH 2 x 10G SDH
Amsterdam
2 x 10G S
DH
2 x 10GBE
2 x 10G SDH
1 x 10G S
DH
1 x 10G SDH 1 x 10G SDH
1 x
10G
SD
H
I2 NewNet
Hybrid Network
Ports at Ports at Each Each PoPPoP
10 10 GbEGbE
OC-192OC-192/STM-64/STM-64
AMSAMS 22 55
GVAGVA 88 77
NYCNYC 44 1212
CHICHI 44 1010
Overcast ;-)Overcast ;-)
LHC OPN Meeting, Munich
Network Forecast - 2010Network Forecast - 2010
Geneva
Chicago
4 x 10GBE
4 x 10GBE
10 x 10GBE
4 x 10G SDH
2 x 10G SDH
ESnet SDN
1 x 10G SDH 1x 10G SDH
3 x 10G SDH
New York
4 x 10G SDH
Amsterdam
3 x 10G S
DH
2 x 10GBE
3 x 10G SDH
1x 10G SDH
1x 1
0G S
DH
1x 10G S
DH
1x 10G SDH
3 x 10G SDH
I2 NewNetHybrid Network
Ports at Ports at Each Each PoPPoP
10 GbE10 GbE OC-192OC-192/STM-64/STM-64
AMSAMS 22 77
GVAGVA 1010 99
NYCNYC 44 1616
CHICHI 44 1212
Severe chances of high-speed data Severe chances of high-speed data transferstransfers
LHC OPN Meeting, Munich
Ciena CD/CICiena CD/CI
- VCAT/LCAS/GFPVCAT/LCAS/GFP support – allows for support – allows for provisioning of dynamic circuitsprovisioning of dynamic circuits
- Difficult to manage with current network tools Difficult to manage with current network tools (no SNMP for example)(no SNMP for example)
- Integrated control plane functionality (still to be Integrated control plane functionality (still to be explored)explored)
- Need to be integrated with our Need to be integrated with our provisioning/monitoring tools (provisioning/monitoring tools (MonALISAMonALISA))
- Also selected by Internet2 for their Also selected by Internet2 for their NewNet NewNet installation (dynamic provisioning with installation (dynamic provisioning with DRAGON software)DRAGON software)
- Inter-domain integration for dynamic circuits is Inter-domain integration for dynamic circuits is desirable in the futuredesirable in the future
LHC OPN Meeting, Munich
Ciena CD/CI (cont.)Ciena CD/CI (cont.)
- Bandwidth granularity of Bandwidth granularity of STS-1STS-1 (51Mbps) (51Mbps)- Bandwidth adjustments Bandwidth adjustments without interrupting the without interrupting the
circuitcircuit (very important for long-lived (very important for long-lived transatlantic data transfers)transatlantic data transfers)
- OSRP + Mesh restorationOSRP + Mesh restoration allows for easy allows for easy provisioning and re-routing of circuits over provisioning and re-routing of circuits over alternate pathsalternate paths
- Fast circuit restoration (Ciena claims 50ms)Fast circuit restoration (Ciena claims 50ms)- Associated circuit priorities (lower priority Associated circuit priorities (lower priority
circuits can be preempted during restoration)circuits can be preempted during restoration)- Need to be integrated with Need to be integrated with perfSONARperfSONAR
monitoring (help needed)monitoring (help needed)
LHC OPN Meeting, Munich
Other Technical IssuesOther Technical Issues
The CD/CI management/monitoring is done only The CD/CI management/monitoring is done only via TL1 or GUI (NodeManager)via TL1 or GUI (NodeManager)
Circuit status is not reflected in the terminating Circuit status is not reflected in the terminating 10 GbE interface status (workaround – UDLD, 10 GbE interface status (workaround – UDLD, FAFD)FAFD)
We need to understand how to interact with the We need to understand how to interact with the CD/CI control planeCD/CI control plane
We need an API to work with for automated We need an API to work with for automated operations; apparently there is a CORBA operations; apparently there is a CORBA interface but we don’t have any specifications interface but we don’t have any specifications for it yet.for it yet.
LHC OPN Meeting, Munich
US LHCNet Network MonitoringUS LHCNet Network Monitoring
Various monitoring tools are deployed Various monitoring tools are deployed - SpectrumSpectrum – used by the CERN operators (first – used by the CERN operators (first
line of support)line of support)- MonALISA MonALISA – used by the US LHCNet team to get – used by the US LHCNet team to get
a global view of all the servicesa global view of all the services- Various open-source toos (Various open-source toos (Nagios, Cricket, Nagios, Cricket,
rancid, RRDrancid, RRD) - used by the US LHCNet NOC for ) - used by the US LHCNet NOC for configuration tracking, logs and alarmsconfiguration tracking, logs and alarms
- perfSONARperfSONAR – used by the E2ECU – used by the E2ECU
Current perfSONAR installation extracts data Current perfSONAR installation extracts data from the Spectrum monitoring service at CERN; from the Spectrum monitoring service at CERN; we have plans to migrate this to a stand-alone we have plans to migrate this to a stand-alone setup (at least for the CD/CIs)setup (at least for the CD/CIs)
LHC OPN Meeting, Munich
US LHCNet TeamUS LHCNet Team
- Harvey NewmanHarvey Newman – Project PI – Project PI- Dan NaeDan Nae – Network Engineer – tech team lead – Network Engineer – tech team lead- Ramiro VoicuRamiro Voicu – Software Engineer – Software Engineer- Two people left – Two people left – Sylvain RavotSylvain Ravot (October, 2006) (October, 2006)
and and Yang XiaYang Xia (March, 2007) (March, 2007) - Artur BarczykArtur Barczyk - new Network Engineer, started - new Network Engineer, started
on March 1on March 1st, st, 2007, based at CERN2007, based at CERN- Tony ChengTony Cheng – new Network Engineer, started – new Network Engineer, started
April 15April 15thth, based in Pasadena, CA, based in Pasadena, CA
LHC OPN Meeting, Munich
Questions?Questions?
Thank You!Thank You!
top related