optiputer infostructure: east of i-5*
DESCRIPTION
OptIPuter Infostructure: East of I-5* Tom DeFanti, Maxine Brown, Jason Leigh, Oliver Yu, Tom Moher, Bob Grossman, Joe Mambretti, Valerie Taylor, Cees de Laat. “East of I-5” OptIPuter. “East of I-5” OptIPuter is focused on large-data applications using experimental technology - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/1.jpg)
OptIPuter Infostructure: East of I-5*
Tom DeFanti, Maxine Brown, Jason Leigh, Oliver Yu, Tom Moher, Bob Grossman, Joe Mambretti, Valerie Taylor, Cees de Laat
![Page 2: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/2.jpg)
“East of I-5” OptIPuter
• “East of I-5” OptIPuter is focused on large-data applications using experimental technology
• Experimental means that – It is obtainable and affordable– It works 99% of the time– It is programmable– Brute force is acceptable
• Our goal: to make photonics controllable by Grid middleware as soon as possible
![Page 3: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/3.jpg)
What is a Lambda?
• A lambda, in networking, is a fully dedicated wavelength of light in an optical network, typically used today for 1-10Gbps.
• We are now working with 1Gb dedicated layer2 circuits that act like lambdas
• We need enough to schedule and manipulate: up to 40 1Gb “sub” lambdas will be available to the OptIPuter locally, regionally, internationally
• We expect 10Gb lambdas to be available to the OptIPuter in a few years; first locally, then regionally, then (inter)nationally
![Page 4: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/4.jpg)
Gross Optical Burst Switching (GOBS)
• Move a terabyte or petabyte on a schedule• Roam a 375600 x 375600 pixel remote database• Applications will be able to request dedicated
Lambdas using the routed infrastructure• Bypass the routers in between
PC
PC
PC
PC
PC
PC
Photonic Switch(Glimmerglass, Calient)
![Page 5: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/5.jpg)
Cluster Visualization
5x3 Grid of 1280x1024 Pixel LCD Panels Driven by 16-PC Cluster Resolution=6400x3072 Pixels, or ~3000x1500 pixels in Autostereo
![Page 6: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/6.jpg)
NTT 4Kx2K Compressed Video from Chicago to Los Angeles
• Pre-compressed to 300 Mbps in Chicago using an experimental JPEG 2000 SHD codec
• Received in LA at USC Zemeckis Center by an NTT real-time decoder, and fed to NTT's prototype SHD frame-buffer and 8-megapixel full-color D-ILA projector for display on a large screen.
• SHD= 4xHDTV or 16xDVD
http://www.ntt.co.jp/news/news02e/0211/021113.html
![Page 7: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/7.jpg)
• Lambdas for high bandwidth applications:– Bypass production network– Middleware request for
optical pipes• Rationale:
– Lower the cost of transport per packet
Application
Middleware
Transport
Application
Middleware
Transport
Router
Router
UvA
Router
Router
CA*net4
Router
ams
chi
SURFnet5
UBC Vancouver
Switch
GbE
GbE
GbE
Lambda
Switch
Lambda
Switch
Switch
Router
High bandwidth application
![Page 8: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/8.jpg)
Scale 2-20-200
![Page 9: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/9.jpg)
EVLEVL
![Page 10: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/10.jpg)
OMNInet Chicago 2x10GE LambdaGrid Scale: 2ms
• A four-site network in Chicago -- a 10GE service trial• A test bed for all-optical switching and advanced middleware• Partners: SBC, Nortel, iCAIR at Northwestern, EVL, CANARIE, ANL
2x10GE
Northwestern U
OpticalSwitchingPlatform
Passport8600
ApplicationCluster
ApplicationCluster
OpticalSwitchingPlatform
Passport8600
2x10GE
StarLight
OPTera Metro5200
ApplicationCluster
OpticalSwitchingPlatform
Passport8600
2x10GE8x1GE
UIC
Carrier Hotel
OpticalSwitchingPlatform
Passport8600
2x10GE8x1GE
8x1GE
8x1GELoop back
![Page 11: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/11.jpg)
UIUC/NCSA
Starlight(NU-Chicago)Argonne
UChicagoIIT
UIC
Illinois Century NetworkJames R. Thompson CtrCity HallState of IL Bldg
4 pair
12 pair
4 pair
2 pair 2 pair
4 pair
18 pair
4 10 pair
12 pair
2 pair
Level(3)111 N. Canal
McLeodUSA151/155 N. MichiganDoral Plaza
Qwest455 N. Cityfront
UC Gleacher450 N. Cityfront
Illinois’ I-WIRE
![Page 12: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/12.jpg)
TransPAC/APAN
Euro-LinkSurfNET
Euro-LinkCERN
GÉANT
CA*net4
Abilene
Euro-Link Funded 10Gb LinkEuro-Link Co-Funded 10Gb LinkTransPAC Co-Funded 10Gb LinkPartner Matching 10Gb LinkGÉANT Matching 2.5Gb Links
Euro-Link/Trans-Light andTransPAC/Trans-Light SupplementFunded, Co-Funded and Matching Links
I-5/PLR
![Page 13: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/13.jpg)
• A Production Network 1GigE and 10GigE exchange• An Experimental Network lambda exchange• A Research Network 1GigE and 10GigE MEMS-switched
exchange• Host to DTFnet, the TeraGrid’s 4x10Gb T640-based
Experimental Network, perhaps for future collaborations• A co-location space with 66 racks for networking and
computing and data-management equipment • An OIX with fiber and/or circuits from SBC/Ameritech,
Qwest, AT&T, Global Crossing, Looking Glass Networks, Level 3, RCN, Deutsche Telekom/T-Systems, I-WIRE
• A facility for links coming from NetherLight, CERN/DataTAG, CA*net4, and proposed from UK-Light and APAN forming Trans-Light
StarLight supports the OptIPuter as
![Page 14: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/14.jpg)
Clusters East of I-5
• Each site (StarLight, UIC, NetherLight) has – Several clusters with dual processors and dual GigE
cards– Electronic switching and routing– Optical switching– Specialized Clusters
• Computing • Data Mining and Serving• Visualization
• Upgrade to Itanium Clusters in progress• Upgrade to 10GigE NICs next year
![Page 15: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/15.jpg)
Lambdas East of I-5, 2003
• Illinois– 16 GigEs to UIC– 8 GigEs to NorthWestern/Evanston– GigEs to ANL, NCSA
• Canada– 8 GigEs Chicago to NYC– 8 GigEs Chicago to Seattle
• Europe– 16 GigEs Chicago to Amsterdam– 4 GigEs Chicago to CERN– 2 GigEs CERN to Amsterdam
![Page 16: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/16.jpg)
Optical MEMS Switching
![Page 17: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/17.jpg)
Why Optical Switching?• No need to look at every packet when transferring
a terabyte of information– 1% the cost of routing– 10% the cost of switching– 64x64 10Gb:
• $100,000 O-O-O switched• $1,000,000 O-E-O switched• $10,000,000 O-E-O Routed
• Spend the savings on links, computing and collaboration systems instead!
• Replaces patch panels; allows rapid reconfiguration of 1 and 10Gb experiments
![Page 18: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/18.jpg)
UIC Procurement of 3D MEMS Switches
• Sent out bid request for (2) 64x64 3D MEMS Switches
• GlimmerGlass Networks and Calient switches were tested with a small cluster– Using both MMF (with adapters) and SMF NICs– Computer controlled– No problems encountered
• Calient won the bid for 2 switches; UIC is upgrading one switch to 128x128
• GlimmerGlass Networks will provide EVL a loaner 64x64 switch
![Page 19: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/19.jpg)
Optical Switches at StarLight and NetherLight
N E T H E R L I G H T
A “groomer” is a box that accepts multiple circuits of varying types (e.g., 1GigE, 10GigE) and aggregates and/or disseminates over the 10Gbps transoceanic link. As the amount of transoceanic connectivity increases, we aim to “bandwidth match” the amount of data being sent and/or received by clusters across continents.
GigE = Gigabit Ethernet (Gbps connection type)
8-processor cluster
16-processor cluster
Switch/Router
8 GigE16 GigE
8 GigE16 GigE
Control plane
Data plane
“Groomer” at StarLight
8 GigE
2 GigE
128x128MEMS
Optical Switch
N-processor cluster
8 GigE
N GigE
“Groomer” at NetherLight
Control plane
Data plane
2 GigE
OC-192
(10Gbps)
N GigE
64x64MEMS
Optical Switch
Switch/Router
![Page 20: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/20.jpg)
SL 650916-processor cluster
8-processor cluster
ONI DWDM
SL OPSW
NL 15454
16
8
2
6
6
2
8
OC-192
128x128 MEMSOptical Switch
OMNInet8
4
10Gb
10Gb
Layer2/3 Switch/Router
EVL 6509
16-processor cluster
16
16
EVL OPSW
64x64 MEMS
16
ONI DWDM
OMNInet
6
8
2
Layer2/3 Switch/Router 16
UIC/EVL LAN
StarLightStarLight LAN
NetherLight/UvA
NL 15454
NL OPSWNL 6509
Layer2/3 Switch/Router
6
2
8-processor cluster
8
8
Fiber Use in Pairs
GE ElectronicallySwitched
GE OpticallySwitched
64x64 MEMSOptical Switch
OptIPuterEVL/StarLight/NetherLight/UvA
Summer 2003 Plan
![Page 21: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/21.jpg)
Hard Infostructure Problems
• Internet is not designed for single large-scale users—TCP is not usable for long fat applications
• Circuits are not scalable, but neither are router$• All intelligence has to be on the edge• Tuning compute, data, visualization, networking
using clusters to get order of magnitude improvement
• Security at 10Gb line speed
![Page 22: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/22.jpg)
Thanks to…• StarLight planning, research, collaborations, and outreach
efforts are made possible, in major part, by funding from: – National Science Foundation (NSF) awards ANI-9980480, ANI-9730202,
EIA-9802090, EIA-9871058, ANI-0225642, and EIA-0115809– NSF Partnerships for Advanced Computational Infrastructure (PACI)
cooperative agreement ACI-9619019 to NCSA– State of Illinois I-WIRE Program, and major UIC cost sharing– Northwestern University for providing space, engineering and
management• NSF/CISE/ANIR and DoE/Argonne National Laboratory for StarLight and
I-WIRE network engineering and planning leadership• NSF/CISE/ACIR and NCSA/SDSC for DTF/TeraGrid/ETF opportunities• The OMNInet Initiative• UCAID/Abilene for Internet2 and ITN transit; IU for the GlobalNOC • Bill St. Arnaud of CANARIE, Kees Neggers of SURFnet, Olivier Martin of
CERN, Michael McRobbie of IU, and Harvey Newman of CalTech
![Page 23: OptIPuter Infostructure: East of I-5*](https://reader036.vdocument.in/reader036/viewer/2022062315/56815dfd550346895dcc38cf/html5/thumbnails/23.jpg)
OptIPuter: East of I-5