The International Grid Testbed: a 10 Gigabit Ethernet success story
in memoriam Bob Dobinsonin memoriam Bob Dobinson
GNEW 2004, Geneva
Catalin Meirosuon behalf of the IGT collaboration
GNEW 2004 2
International
Testbed Contents
• www: who, why, what is IGT
• Native Ethernet over lightpaths: from 1 Gbps to 10 Gbps
• First demonstration of transatlantic native 10 GE over legacy OC-192 infrastructure
• Future plans
• Conclusions
GNEW 2004 3
International
Testbed Who’s who in IGT• Universities and Research Institutes
– Canada• Carleton University, Ottawa• University of Alberta• University of Victoria• TRIUMF, Vancouver• Université de Montreal• McGill University, Montreal• University of Toronto
– The Netherlands• University of Amsterdam
– Switzerland• CERN, Geneva
• Connectivity provided byCANARIE SURFnet ORION Netera BCnet RISQ
• Sources of funding: IGT (Canarie Directed Research Grant), ESTA (IST-2001-33182)
• Collaboration: CERN Openlab, the EU DataTAG project
GNEW 2004 4
International
Testbed Current IGT topology
GNEW 2004 5
International
Testbed Motivation
• IGT is a research network for next-generation e-science– Illustration of the application empowered
network concept (see Bill St. Arnaud’s panel at SC2003)
– Investigate emerging technologies in a demanding real life environment (see Wade Hong’s presentation at the CA*Net 4 Design Meeting May 26-27, 2003 )
– Real Time Farms for the ATLAS experiment at CERN (see Bryan Caron’s presentation at RISQ2003)
GNEW 2004 6
International
Testbed The ATLAS Experiment at CERN
• Large collaboration of about 2000 scientists for an experiment operational from 2007
• Main goals: the discovery of new particles and exploring physics beyond the Standard Model
• Challenging data collection and analysis systems– Online: 20 Gbps to the
event filter– Offline: 4 TB/day to
storage
GNEW 2004 7
International
Testbed Ethernet: from local to global
• Ethernet, a brand name for LAN technology– Original Ethernet: shared media, half-duplex,
distance limited by protocol– Modern Ethernet: point-to-point, full-duplex,
switched, distance limited by the optical components
• Cost effective !
GNEW 2004 8
International
Testbed Why native Ethernet long haul?• More than 90% of the Internet traffic originates on an Ethernet
LAN• Data traffic on the LAN increases due to new applications• Ethernet services with incremental bandwidth offer new
business opportunities to carriers– See IEEE Communications Magazine, Vol. 42, No. 3, March 2004, on
additional benefits for both the enterprise and the service providers
• Why not native Ethernet ?– Scalability, reliability, service guarantees …
• All of the above are active research areas
• Native Ethernet long haul connections can be used today as a complement to the routed networks, not as a replacement
GNEW 2004 9
International
Testbed 10 GE: a new Ethernet
• 10 GE – designed from the beginning for access to long haul networks– 40 km maximum distance specified by the standard …
– 1550nm lasers: optical amplifiers can be used to increase distance over dark fibre
• State of the art: 250 km demonstrated in Denmark by the EU ESTA project [see Mikkel Olesen’s presentation at NORDunet2003]
– what happens when you have to regenerate the signal ?• no signal-agnostic regenerators deployed
GNEW 2004 10
International
Testbed The 10 GE WAN PHY
• 10GE introduces a gateway from LAN to the WAN by means of the WAN PHY– Compatible with existing
WAN infrastructure• Transmission rate• Encapsulation
– Partial use of the management bits of the SONET/SDH frame
• Today’s WAN PHY modules use SONET-compliant optical components
WANWANPHY
LTE
LTE3R
3R
3R
OC192Router
10GE switch/router
LTE
RouterOC192
WANPHY
10GE switch/router
LTE
traditional
novel
GNEW 2004 11
International
Testbed Demo during ITU Telecom World '03
CiscoONS 15454Force10
E 600
Force10E 600
HPItanium-2
HPItanium-2
CiscoONS 15454
CiscoONS 15454
CiscoONS 15454
CiscoONS 15454
Ixia400T
IntelItanium-2
IntelXeon
Ixia400T
10GE WAN PHY 10GE LAN PHY OC192c
Ottawa Toronto Chicago Amsterdam Geneva
10 GE WAN PHY over an OC-192c circuit using lightpaths provided by SURFnet and CANARIE
9.24 Gbps using traffic generators
5.65 Gbps using TCP on PCs
6 Gbps using UDP on PCs
GNEW 2004 12
International
Testbed Results on the transatlantic 10 GE
Single stream UDP throughput Single stream TCP throughput
•Data rates are limited by the PC, even for our memory-to-memory tests•UDP uses less resources than TCP on high bandwidth-delay product networks
GNEW 2004 13
International
Testbed WAN PHY over DWDM
• Direct lambda access from the provider is required• The DWDM transceiver card as “LTE”
HP Itanium-2
10 GE WAN
DWDM
DWDM
Amsterdam Geneva
Force10 E600
10GE LAN
HPItanium-2
10 GE WAN10GE LAN
Ixia400T
Ixia400T
Force10 E600 HP
Itanium-2
IntelXeon
GNEW 2004 14
International
Testbed What next ?
• More 10 Gigabit Ethernet experiments– Extend the reach of our WAN PHY connection
• Adds more latency into the testbed, hence increased pressure on protocols
• Might allow us to identify technical showstoppers
– Disk-to-disk transfers– Comparative study of data transfer protocols
for 10 Gbps networks
GNEW 2004 15
International
Testbed What next ?
• Support for real-time access to remote computer farms during the ATLAS testbeam run in 2004– 1 Gbps connectivity is sufficient as a proof of
concept– Sites in Copenhagen, Cracow and Edmonton
will process in real time data acquired at the CERN testbeam
• This is part of the feasibility study for remote real time processing in ATLAS
GNEW 2004 16
International
Testbed Conclusions
• IGT demonstrated native 10 Gigabit Ethernet over lightpaths
• 10GE WAN PHY is the technology that enables inter-continental native Ethernet
• IGT will support remote real time applications, with emphasis on data collection and analysis
GNEW 2004 17
International
Testbed References
• Bill St. Arnaud – contribution to the “Strategies for Application-Empowered Networks” panel at SC2003
• Bryan Caron – “The GARDEN Project: A Testbed for High Bandwidth Real-Time Applications”, http://www.risq.qc.ca/risq2003-canw2003/ppt/23PM_Bryan_Caron.pdf
• Wade Hong – “CA*net 4 International Grid Testbed ”, http://lightpath.physics.carleton.ca/C4DesignIGTMay27.ppt
• Mikkel Olesen – “The use of dark fibre in Forskningsnttet”, the 21st NORUnet Conference, http://www.nordunet2003.is/smasidur/presentations/MikkelOlesen.ppt