dan nae california institute of technology us lhcnet update
TRANSCRIPT
Dan NaeDan Nae
California Institute of Technology California Institute of Technology
US LHCNet UpdateUS LHCNet Update
DEN
ELP
ALB
ATL
Metropolitan Area Rings
SEA
Major DOE Office of Science Sites
SNV
MetropolitanAreaRings
10Gb/s
20-30-40-50 Gb/s
20-30-40 Gb/s
NYC
CHI
DC
SDNetwork core, 20-30-40-50 Gbps circuit based transportMajor international
Production IP ESnet core, 10 Gbps enterprise IP traffic
LHCNet Data Network
NSF/IRNC circuit; GVA-AMS connection via Surfnet or Geant2
HOU
TUL
KAN
SDG
LA
ESNet4Science Data Network Core
(20-50 Gbps)
ESnet4IP Core (≥10
Gbps) US LHCNet Data Network
(3x10Gbps to the US)
NYC
BNL
FNAL
CERN
AMS
US LHCNet + ESnet (2007)US LHCNet + ESnet (2007)
2007
Connections to ESnet Hubs in New-York and Chicago Redundant “light-paths” to BNL and FNAL Redundant 10 Gbps peering with Abilene Access to USNet/HOPI for R&D
2006
Four providers:Colt QwestGlobal Crossing GEANT
Atlantic Ocean
Multiple Fiber Paths: Multiple Fiber Paths: Reliability Through DiversityReliability Through Diversity
NYC 111 8th
Pottington (UK)
VSNL South
NY60 Hudson
Highbridge
VSNL North
AMS-SARAAC-2
Bude
GVA-CERN
FrankfurtVSNL
Wal, NJ
London
Global Crossing
Qwest
Colt
GEANT
NYC-MANLAN
CHI-Starlight
Paris
Bellport Whitesands
LCG Availability LCG Availability requirement: 99.95%requirement: 99.95%
Brookhaven
LONAC-1
EASTWEST
Additional SlidesAdditional Slides
Equipment DiversityEquipment Diversity
Single point of failure for the CERN-FNAL
traffic
US LHCNet, ESnet and the two US Tier1s (FNAL and BNL) are working to achieve complete equipment diversity for the primary and backup paths
Equipment Diversity (cont.)Equipment Diversity (cont.)
The new setup allows for independent paths and can survive the failure of any single piece of equipment
Great advantage in case of hardware of software maintenance Similar setup for the CERN-BNL connection
Ciena Transitions (Today)Ciena Transitions (Today)
Two parallel networks, one Force10 and one Ciena Today the main links (CERN-BNL, CERN-FNAL) go over the Force10s (proven
reliability, stable configuration) Circuit oriented services development for the Cienas
GEANT-ESnet Peering, FNAL-GridKa, BNL-
GridKa
Planned Configuration (2008)Planned Configuration (2008)
Robust fallback at layer 1 + next-generation hybrid optical network: Robust fallback at layer 1 + next-generation hybrid optical network: Dynamic Dynamic circuit-oriented network services with BW guarantees circuit-oriented network services with BW guarantees
Emerging Emerging StandardsStandards
VCAT, LCASVCAT, LCAS
Provisioned circuits over a failed SONET link can be re-routed according to priorities and preempt lower priority circuits.
Fallback is automatic and very fast
(<50ms once failure is detected)
Ciena “Mesh Restoration” of a CircuitCiena “Mesh Restoration” of a Circuit
Network ForecastNetwork Forecast
Ports at Each PoP 10 GbE Ports OC-192/STM-64 Ports
AMS 2 7
GVA 10 9
NYC 4 16
CHI 4 12
Geneva
Chicago
4 x 10GBE
4 x 10GBE
10 x 10GBE
4 x 10G SDH
ESnet SDN
3 x 10G SDH
New York
4 x 10G SDH
Amsterdam
2 x 10GBE
3 x 10G SDH
Circuit-enabled regional network
Circuit-enabled regional network
Geneva
Chicago
New York
4 x 10GBE
4 x 10GBE
8 x 10GBE
3 x 10G SDH
3 x 10G SDH
ESnet SDN
2 x 10G SDH 2 x 10G SDH
Amsterdam
2 x 10GBE
Circuit-enabled regional network Circuit-enabled
regional network
Ports at Each PoP 10 GbE Ports OC-192/STM-64 Ports
AMS 2 5
GVA 8 7
NYC 4 12
CHI 4 10
2009 2010
Double density cards and matrix
40 Gbps wavelengths?
Two port 10 GbE card
LHCNet connections to ESnet: FY09/FY10LHCNet connections to ESnet: FY09/FY10
NLR wavegear sites
NLR regeneration / OADM sites
ESnet via NLR (10 Gbps waves)LHCNet (10 Gbps waves)
Denver
Seattle
Su
nn
yva
le
LA
San Diego
Chicago
PittsWash DC
Raleigh
Jacksonville
Atlanta
KC
Baton Rouge
El Paso - Las Cruces
Phoenix
Pensacola
Dallas
San Ant.Houston
Albuq. Tulsa
New YorkClev
Boise
CERN (Geneva)
LHCNet: To ~80 Gbps by 2009-10Routing + Dynamic managed
circuit provisioning
Network MonitoringNetwork Monitoring
MonALISA (TL1 Module) Spectrum (CERN first line) Various open source tools
(cricket, nagios, rancid, syslog-ng, etc)
perfSONAR (GEANT E2ECU)
True end-to-end (host-to-host) monitoring using MonALISA
“Network intelligence” or the ability to reconfigure the circuits based on performance, changing network conditions or high-priority scheduled transfers
US LHCNet Working Methods US LHCNet Working Methods
Production NetworkDevelop and build next generation networks
Networks for ResearchD0, CDF, BaBar, CMS, Atlas
GRID applications PPDG/iVDGL, OSG, WLCG,
DISUN
LHCOPN
Interconnection of US and EU Grid domains
VRVS/EVO
High performanceHigh bandwidthReliable network
HEP & DoE Roadmaps
Testbed for Grid Development
Pre-ProductionN x 10 Gbps transatlantic testbed
New Data transport protocols
Interface and kernel setting
HOPI / UltraScience Net / Ultralight / CHEPREO /
LambdaSation
Lightpath technologies
Vendor Partnerships
Four Continent Testbed and FacilityFour Continent Testbed and Facility
Building a global, network-aware end-to-end managed real-time Grid
Caltech, Florida, FIU, UMich,
SLAC,FNAL, CERN, Internet2, NLR, UERJ(Rio),
USP, CENIC, Starlight, Cisco
http://ultralight.caltech.eduhttp://ultralight.caltech.eduhttp://ultralight.caltech.eduhttp://ultralight.caltech.edu
Network Services for Managed Network Services for Managed End-to-End Data TransfersEnd-to-End Data Transfers
Robust Network Services based on
Bandwidth guarantees Virtual Circuits
Scheduled Transfers Transfer Classes Priorities
Monitoring of all components end-to-end Network Elements End-Hosts
Interface to other circuit-oriented systems Be part of
heterogeneous end-to-end infrastructure
Problem Finding and Resolution Problem Finding and Resolution Problems encountered today are
hard to track due to missing the global view of the system
Example situation: the system recognizes an end-host problem during the transfer and takes mitigating actions, re-scheduling transfers and notifying operators
Developed in the field-proven MonALISA Framework
End-to-end MonitoredManaged Transfers
Track problem to the source Network / End-host
Take appropriate action Change transfer path Adjust end-host parameters Re-schedule transfer
Provide experts with relevant (real-time) information Keep the user/application up-to-date on transfer progress Progessive automation: Target
optimal resource utilization
Tom Lehman, GLIF 2007 Winter Workshop
http://www.glif.is/meetings/2007/winter/controlplane/lehman-dynamic-services.pdf
GLIFConcep
t
US LHCNet MilestonesUS LHCNet Milestones
““Pre-production”:Pre-production”: The new infrastructure initially deployed The new infrastructure initially deployed 2007 will offer circuit-based services intended to provide 2007 will offer circuit-based services intended to provide redundant paths and on-demand, high bandwidth end-to-end redundant paths and on-demand, high bandwidth end-to-end dedicated circuits. Circuit-switched services will be used to dedicated circuits. Circuit-switched services will be used to directly interconnect the DOE laboratories to CERN and will be directly interconnect the DOE laboratories to CERN and will be available on demand to policy-driven, data-intensive available on demand to policy-driven, data-intensive applications, managed by MonALISA servicesapplications, managed by MonALISA services
End of 2007:End of 2007: initial deployment of our circuit oriented network initial deployment of our circuit oriented network services on US LHCNet; simple scheduler with fixed bandwidth services on US LHCNet; simple scheduler with fixed bandwidth circuits for site to site on-demand data set transfers.circuits for site to site on-demand data set transfers.
Spring 2008:Spring 2008: interaction with the data transfer application of the interaction with the data transfer application of the experiments, as well as with other intra-domain and inter-experiments, as well as with other intra-domain and inter-domain (LambdaStation, TeraPaths, DRAGON, Oscars) control domain (LambdaStation, TeraPaths, DRAGON, Oscars) control plane services in order to provide an end-to-end path plane services in order to provide an end-to-end path reservation.reservation.
LHC Startup:LHC Startup: July 2008: July 2008: We will begin to exercise the network We will begin to exercise the network and services with real data, in close cooperation with the LHC and services with real data, in close cooperation with the LHC experiments. experiments.