Translight/PacificWaveTranslight/PacificWave
John SilvesterJohn SilvesterExecutive Director of Center for Scholarly TechnologyExecutive Director of Center for Scholarly TechnologyProfessor or Computer Engineering, USCProfessor or Computer Engineering, USCChair, CENICChair, CENIC
Briefing to NSF-OISE, April 28, 2006
NSF OCI-IRNC Cooperative Agreement [#0441119]NSF OCI-IRNC Cooperative Agreement [#0441119]
Translight/PacificWave (TL/PW)Translight/PacificWave (TL/PW) Cooperative Agreement [#0441119] under NSF Cooperative Agreement [#0441119] under NSF
IRNC Program IRNC Program PI John Silvester (USC); Co-PI Ron Johnson PI John Silvester (USC); Co-PI Ron Johnson
(UW)(UW) Other Key PartnersOther Key Partners
CENIC, Jim DolgonasCENIC, Jim Dolgonas PNWG, Jacqueline BrownPNWG, Jacqueline Brown University of Hawaii, David LassnerUniversity of Hawaii, David Lassner AARnet, George McLaughlinAARnet, George McLaughlin
Approx $1M per year for 5 yearsApprox $1M per year for 5 years Awarded March 1, 2005Awarded March 1, 2005
TL/PW Primary ObjectivesTL/PW Primary Objectives Facilitate international R&E connections on US Pacific Facilitate international R&E connections on US Pacific
CoastCoast Assist in the termination (equipment, backhaul) of Assist in the termination (equipment, backhaul) of
AARnet SXTransport links to Hawaii, Seattle and Los AARnet SXTransport links to Hawaii, Seattle and Los AngelesAngeles
Assist in build-out of Hawaii connectivity to TelescopesAssist in build-out of Hawaii connectivity to Telescopes Further develop PacificWave exchange capabilitiesFurther develop PacificWave exchange capabilities Assist in operation of IEEAF link Tokyo-SeattleAssist in operation of IEEAF link Tokyo-Seattle Assist in operation and support of other International Assist in operation and support of other International
R&E networks participating in PWR&E networks participating in PW Develop and operate advanced capabilities to support Develop and operate advanced capabilities to support
optical interconnect and exchange needs of R&E optical interconnect and exchange needs of R&E networksnetworks
What is Pacific Wave?What is Pacific Wave?
Pacific Wave is a state-of-the-art international Pacific Wave is a state-of-the-art international peering facility designed to serve advanced peering facility designed to serve advanced research and education.research and education.
Primary focus is interconnectivity among US Primary focus is interconnectivity among US R&E networks and International R&E networks R&E networks and International R&E networks from the Pacific Rimfrom the Pacific Rim
One of the objectives was the desire for all One of the objectives was the desire for all participants to be able to peer with each other participants to be able to peer with each other which led to the “distributed exchange” designwhich led to the “distributed exchange” design
Context of Pacific WaveContext of Pacific Wave
The US States on the Pacific Coast [Alaska, The US States on the Pacific Coast [Alaska, Washington, Oregon, California, Hawaii] have Washington, Oregon, California, Hawaii] have collaborated in networking for several years. (e.g. US collaborated in networking for several years. (e.g. US Pacific Consortium is an affiliate of APAN)Pacific Consortium is an affiliate of APAN)
Seattle, Los Angeles, and Sunnyvale are “natural” places Seattle, Los Angeles, and Sunnyvale are “natural” places for exchange pointsfor exchange points
Many international links enter the US through Many international links enter the US through Washington and California (and to a certain extent Washington and California (and to a certain extent Hawaii although they tend to remain local there)Hawaii although they tend to remain local there)
Led to the development of Pacific Wave – a collaboration Led to the development of Pacific Wave – a collaboration between CENIC and PNWG to build out exchange between CENIC and PNWG to build out exchange capabilities on West Coast.capabilities on West Coast.
Who Operates Pacific Wave?Who Operates Pacific Wave?A joint project of CENIC and Pacific Northwest Gigapop
In collaboration with University of Southern California and University of Washington
Pacific Wave Layer 2 ExchangePacific Wave Layer 2 Exchange Extensible peering exchange to allow layer 3 networks to peer easily Extensible peering exchange to allow layer 3 networks to peer easily
over a layer 2 fabricover a layer 2 fabric Nodes (currently) in Seattle, Sunnyvale, and Los Angeles. Connected Nodes (currently) in Seattle, Sunnyvale, and Los Angeles. Connected
by a 10GbE wave provisioned over National LambdaRail (2,241 by a 10GbE wave provisioned over National LambdaRail (2,241 kilometers)kilometers)
AUP freeAUP free Supports IPv4 and IPv6; multicast enabledSupports IPv4 and IPv6; multicast enabled Provides 24x7 NOC support.Provides 24x7 NOC support. Priced consistently from node to node Priced consistently from node to node Allows participants to self-select their peeringAllows participants to self-select their peering Allows participants to connect to one-location and access participants Allows participants to connect to one-location and access participants
at all Pacific Wave nodesat all Pacific Wave nodes Supports advance applicationsSupports advance applications Welcomes any research or development network that can meet the Welcomes any research or development network that can meet the
minimum network configuration requirements (connect with a router; minimum network configuration requirements (connect with a router; support BGP; support jumbo-frames)support BGP; support jumbo-frames)
““Layer 3” R&E Networks and Pacific WaveLayer 3” R&E Networks and Pacific Wave
AARNET
GEMNET
GLORIAD
IEEAF
KREONET
MIMOS (Malaysia)
QATAR
SINET
SINGAREN
TANET/TWAREN
TRANSPAC2
ABILENE (Internet2)
CALREN (HPR)
DREN ESNET
NASA
NLR-Packetnet
PNWG
Pacific Wave
CA*NET4
CLARACUDI
PacificWave SeattlePacificWave Seattle
Seattle WestinCISCO 6509
NLR Lambda to LAX (via Sunnyvale)
Abilene
AARNet
CA*NET4
GEMnet
SingAREN
TANET2/TWARENATTBI/Comcast
DREN
KREONet2
ESnet Microsoft
PNWGP
NLR Lambda to STARLIGHT
WIDE/TLEX(IEEAF)
NLR-Packetnet
Does not show “lightpath” connections
1 Gigabit
10 Gigabit
Pacific Wave - SunnyvalePacific Wave - Sunnyvale
NLR TO SEATTLE PWAVE
L3 SVL6509
CENIC SVL HPR
NLR TO LA PWAVE
PAIX
ESNET
Does not show “lightpath” connections
1 Gigabit
10 Gigabit
Pacific Wave - Los AngelesPacific Wave - Los Angeles
1 Wilshire6509
600 W 7th6509
818 W 7th6509
CalREN-HPR
10 GE NLR LambdaTo Seattle PWave(via Sunnyvale)
Abilene
TWAREN
TRANSPAC2
Los Nettos
Qatar Foundation
NII/SINET
AARNet
(3)Singaren
Mimos Berhad(Malaysia)
Cinegrid NLRPacketnet
Does not show “lightpath” connections
1 Gigabit
10 Gigabit
IRNC TL/PW Pacific ConnectionsIRNC TL/PW Pacific Connections
CA*Net4 POP
TλEX Tokyo PW-Seattle
AARnet POP Sydney
Hawaii
OahuPW-LA
CLARA, CUDI POPs (Tijuana)
AARnet-SX Transport
IEEAF Link
TL-PW – West Coast DetailTL-PW – West Coast Detail
AARNet-SX(N)To Kahe Point (HI)
Los Osos(CA)
SLO(CA)
Hillsboro(OR)
Portland (OR)
PW-SeattleNLR link
PW-Los Angeles
Pacific WaveN-S link
AARNet-SX(S)To Spencer Beach (HI)
SEA
LAX
Santa Barbara(CA)
SNV
(NLR)
(CalREN-XD)
Optics to carrynew lambda
over CalREN-XD from SLO to LA
TL/PW – Hawaii DetailTL/PW – Hawaii Detail
SCCNOC-192
SCCNOC-192
AARNet-SX(S)To Sydney
Spencer Beach(Big Island)
Mauna KeaMKOCN (Summit)
Hilo
AARNet-SX(S)To Los Osos
Waimea
Kahe Point(Oahu)
Hawaii Gigapop(Oahu)AARNet-
SX(S)To Sydney
AARNet-SX(N)To Portland
Mauna Kea Observatories Communication Network
CSO Gemini IRTFJCMT SMA UKIRT UH-2.2 UH-0.6 VLBACFHT Keck
CSO
Gemini
JAC
Subaru
SMA
CFHT
Keck
VLBA
UH IfA Hilo
UH-Managed Shared Network for All Observatories
Verizon Summit
Verizon Hilo
VerizonHumuula
VerizonWaimea
U N I V E R S I T YU N I V E R S I T Y
UH Hilo Campus
Subaru
DS3
Leased Private Links
Carrier Infrastructurefor leased shared & private links
Leased DS3 (shared)
UH managed fiber links
56kb
Hilo Base Facilities
Waimea Base Facilities
Proposed New GbE
Status of TL/PW as of April 2006Status of TL/PW as of April 2006
Engineering and technical support through PNWG Engineering and technical support through PNWG and CENIC for all International connections into and CENIC for all International connections into Pacific WavePacific Wave
Northern AARnet LinkNorthern AARnet Link Operational and in useOperational and in use
Southern AARnet LinkSouthern AARnet Link Backhaul from Los Osos through San Luis Obispo to PW-Backhaul from Los Osos through San Luis Obispo to PW-
LA over CALREN as OC-192 in placeLA over CALREN as OC-192 in place ETE equipment for HI and LA on order (arriving as we ETE equipment for HI and LA on order (arriving as we
speak).speak). Local connectivity in Hawaii – in processLocal connectivity in Hawaii – in process Expected operation (test mode) end of MayExpected operation (test mode) end of May
Changing Requirements for Changing Requirements for ExchangesExchanges
The Exchange points need to provide a wider The Exchange points need to provide a wider array of services:array of services: Layer 3 Interconnect (routed)Layer 3 Interconnect (routed) Layer 3 Interconnect (over layer 2 switch)Layer 3 Interconnect (over layer 2 switch) Layer 2 Interconnect (Ethernet switching)Layer 2 Interconnect (Ethernet switching) Layer 1 Interconnect (wave switching)Layer 1 Interconnect (wave switching) Layer 0 Interconnect (fiber interconnect)Layer 0 Interconnect (fiber interconnect)
PacificWave is moving in this direction to allow PacificWave is moving in this direction to allow us to continue to participate in the world of us to continue to participate in the world of Global R&E networkingGlobal R&E networking
Pacific Wave Lightpath InterconnectPacific Wave Lightpath Interconnect
Primary focus has been on Layer 3 interconnectPrimary focus has been on Layer 3 interconnect There is a lot of interest in Layer 2 (“lightpaths”) There is a lot of interest in Layer 2 (“lightpaths”)
and Layer 1 (“waves” and “lambdas”) and Layer 1 (“waves” and “lambdas”) interconnect. [GLIF, etc]interconnect. [GLIF, etc]
SXTransport(S) will be configured as 8x1GESXTransport(S) will be configured as 8x1GE PW is deploying additional OC192 from LA to PW is deploying additional OC192 from LA to
SEA configured as 8x1GE to allow interconnect SEA configured as 8x1GE to allow interconnect for AARnet and other nets as demand grows.for AARnet and other nets as demand grows.
Also plans in place to extend PW connectivity to Also plans in place to extend PW connectivity to Starlight (Chicago) to further enhance layer 3 Starlight (Chicago) to further enhance layer 3 and lightpath interconnect capabilityand lightpath interconnect capability
Pacific Wave Lightpath ExchangePacific Wave Lightpath ExchangeTo Canarie
NLR To Chicago and Starlight
San Diego
NLR To Phoenix
IEEAF to TλEX
AARnet to AU, via HI
HOPI
HOPI
SEA
SNY
LA
Lightpath Networks and PWLightpath Networks and PW
Others in the planning stages
TRANSPAC (?)
AARNET (S)
(to TλEX) IEEAF
CA*NET4
NLR Waves
NLR (Layer 2)
CALREN-XD
HOPI (Internet2)
PNWG
Pacific Wave
AARNet S PathPW Lightpath Interconnect
Abilene
TwarenSingaren
PW-L3X:10GLAX-SNY
PW-L2X (8x1G)LAX-SNY-SEA
AARnetRouter
PW 15454
CalRENRouter
6509818
6509
6509
PW-L3X-LA
to SNY, SEA
CENIC 15808
to SLO
to SD
CENIC 15808
CENIC 15808
NLR Packetnet
ActivitiesActivities
Basically all International R&E traffic to Basically all International R&E traffic to Pacific Rim flows through PWPacific Rim flows through PW
Recent HighlightsRecent Highlights iGRIDiGRID SC’05SC’05 PRAGMAPRAGMA APANAPAN INTERNET2 meetingsINTERNET2 meetings
ExamplesExamples January 2005: Huygens Titan probe data transfer: January 2005: Huygens Titan probe data transfer:
3Tbytes of satellite data distributed and collated for 3Tbytes of satellite data distributed and collated for analysis to locations in Amsterdam and Australiaanalysis to locations in Amsterdam and Australia
January 2005: Tele-microscopy demo between January 2005: Tele-microscopy demo between University of Hawaii and UCSD (Lariat network)University of Hawaii and UCSD (Lariat network)
January 2005: PTC Honolulu 270Mbps interactive HD January 2005: PTC Honolulu 270Mbps interactive HD video conferencing with Australia and Seattlevideo conferencing with Australia and Seattle
January 2005: Seattle—Tokyo, uncompressed HDTV 1.5 January 2005: Seattle—Tokyo, uncompressed HDTV 1.5 Gbps Smarr HD lecture between Seattle and JGN Gbps Smarr HD lecture between Seattle and JGN meeting meeting
June 2005: Chicago-Seattle-UCSD OptiPuter HD June 2005: Chicago-Seattle-UCSD OptiPuter HD permanent interactive uncompressed HDTV video-permanent interactive uncompressed HDTV video-conference linkconference link
Examples – iGRID05Examples – iGRID05 4k interactive HD between Tokyo & SD4k interactive HD between Tokyo & SD Concurrent n-way real-time low-latency uncompressed Concurrent n-way real-time low-latency uncompressed
1080 line (ie >1gb/s per stream) HDTV teleconferencing 1080 line (ie >1gb/s per stream) HDTV teleconferencing between Australia, Japan, Korea, Ann Arbor, Seattle between Australia, Japan, Korea, Ann Arbor, Seattle and San Diego and San Diego
100 megapixel remote viz. Illinois-SD100 megapixel remote viz. Illinois-SD Real-time flat-panel stereo, remotely rendered Real-time flat-panel stereo, remotely rendered Clusters in Chicago & Amsterdam loosely coupled Clusters in Chicago & Amsterdam loosely coupled
compute demo: performance = to single machine room compute demo: performance = to single machine room alternative!!alternative!!
Neptune/‘Looking’ real-time HDTV from 2km below the Neptune/‘Looking’ real-time HDTV from 2km below the surface showing volcanic vents in the northeast pacificsurface showing volcanic vents in the northeast pacific
ExamplesExamples
November 14-17, 2005: SC05 Seattle. About 50 10gb/s November 14-17, 2005: SC05 Seattle. About 50 10gb/s waves, petabytes of storage, leading edge visualization waves, petabytes of storage, leading edge visualization systems and teraflops of computing assembled for a systems and teraflops of computing assembled for a week computational sciences etc conference for this. week computational sciences etc conference for this.
‘‘Persistent’ OptIPuterPersistent’ OptIPuter Wide Area Storage GRIDsWide Area Storage GRIDs The Next Disaster Recovery Architectures that really The Next Disaster Recovery Architectures that really
work!? work!? Neptune Canada ‘in the water’ Neptune Canada ‘in the water’
Recent Stereo VideoRecent Stereo Video
I2 Spring MM from Stanford Medical School (Margaret I2 Spring MM from Stanford Medical School (Margaret Krebs, Testbed Mgr, Stanford University Medical Media Krebs, Testbed Mgr, Stanford University Medical Media and Information Technologies) ran a stereo video and Information Technologies) ran a stereo video program test with CSIRO in Sydney on March 30th via program test with CSIRO in Sydney on March 30th via Abilene and SX. Abilene and SX.
They ran about a 90Mbps stream for the test. They ran about a 90Mbps stream for the test. No problems through AARnet N and across AARnet.No problems through AARnet N and across AARnet.
iGrid 2005, San Diego Sept. iGrid 2005, San Diego Sept. 20052005
iGrid 2005, Over 10 Gbps bidirectional traffic coexisted iGrid 2005, Over 10 Gbps bidirectional traffic coexisted with production exchange traffic without detrimentwith production exchange traffic without detriment
4k line interactive Super-HD between Keio University and UCSD4k line interactive Super-HD between Keio University and UCSD Live HD from the sea floor 100 miles off the Pacific CoastLive HD from the sea floor 100 miles off the Pacific Coast N-way uncompressed multicast HD video conferencingN-way uncompressed multicast HD video conferencing
ResearchChannel N-Way HD ResearchChannel N-Way HD Multicast Video ConferencingMulticast Video Conferencing
Does it Really Perform in Does it Really Perform in Practice? …Yes!Practice? …Yes!
7.5 Gbps of unicast 7.5 Gbps of unicast HDTV over IP traffic HDTV over IP traffic (10 750 Mbps flows)(10 750 Mbps flows)
3 Gbps of multicast 3 Gbps of multicast replicated in Seattle replicated in Seattle switch node using switch node using PIM snoopingPIM snooping
Traffic test performed Traffic test performed in Cisco POC lab in Cisco POC lab before the eventbefore the event
HDTV Conferencing Spanning Two HDTV Conferencing Spanning Two Oceans Enabled by Pacific Wave Oceans Enabled by Pacific Wave
Traffic Flows at SupercomputingTraffic Flows at Supercomputing
Among other things, 5-way multicast Among other things, 5-way multicast replication of 3 Gbps, over 12 Gbps total replication of 3 Gbps, over 12 Gbps total traffic across the switch backplanestraffic across the switch backplanes