The OptIPuter Project

Prepared For:Montana State University

Telcordia Contact:George Clapp732-758-4346clapp@research.telcordia.comMonday, October 20, 2003

An SAIC Company

MSU 10/20/2003 – 2

Overview

OptIPuter: combination of Optical, IP, and computerObjective: Distributed Virtual Computer– “…enable collaborating scientists to interactively explore massive

amounts of previously uncorrelated data by developing a radical new architecture…”

– “…’waste’ bandwidth and storage in order to conserve ‘scarce’ computing…”

– “…tightly-integrated cluster of computational, storage, instrumentation and visualization resources, linked over parallel optical networks…”

MSU 10/20/2003 – 3

Fundamental Premise:Mismatch in Rates of Growth

Scientific American January 2001

Computation is the scarce resourceElectronic packet processing is the bottleneck

MSU 10/20/2003 – 4

OptIPuter Organization

National Science Foundation (NSF) Information Technology Research (ITR) program

– Directorate for Computer and Information Science and Engineering (CISE)– $13.5M over 5 years– Began October 2002– First annual review on September 29, 2003– http://www.optiputer.net/events/9-29-03.html– Slides are credited to the researchers

Participants– University of California-San Diego (L. Smarr, A. Chien, G. Hidley, M.

Ellisman)– San Diego Supercomputer Center (P. Papadopoulos)– San Diego State University (E. Frost)– University of California-Irvine (R. Grossman)– University of Southern California (J. Bannister, C. Kesselman)– University of Illinois at Chicago (T. DeFanti, M. Brown, J. Leigh, O. Yu)– Northwestern University (J. Mambretti)– Apologies to those not mentioned

MSU 10/20/2003 – 5

OptIPuter Motivation and Objectives

Motivation from e-Science– Grid– National Cyberinfrastructure

What are the Science Barriers We are Trying to Overcome?– Scientists need collaborative and visual interactions with gigabyte

data objects– Production, Shared Internet Limits Speed of File Transfers– Inadequate Campus Grid Infrastructure

Creating OptIPuter Laboratories at San Diego and ChicagoSystem Software: From Grid to LambdaGridEducation and Outreach

MSU 10/20/2003 – 6

Gigabyte Data Objects Need Interactive Visualization

Extremely large data sources– E.g., microscopes and telescopes– Remote Sensing– Seismic or Medical Imaging– Supercomputer Simulations

Hundred-Million Pixel 2-D Images require new presentation techniques

MSU 10/20/2003 – 7

On-Line Microscopes Create Very Large Biological Montage ImagesLaser Microscope– High Speed On-line

CapabilityUsing High-Res IBM Displays to Interactively Pan and Zoom Large Montage ImagesMontage Image Sizes Exceed 16x Highest Resolution Monitors– ~150 Million Pixels!

Source: David Lee, NCMIR, UCSD

IBM 9M Pixels

MSU 10/20/2003 – 8

NSF’s EarthScopeRollout Over 14 Years Starting

With Existing Broadband Stations

MSU 10/20/2003 – 9

Many Groups Are Experimenting with Tiled DisplaysFor 2D-Montage and 3D-Volumetric Viewing

PerspecTile Running JuxtaView Jason Leigh, EVL, UIC

www.llnl.gov/icc/sdd/img/images/pdf/Walls97.pdf

LCD Panels

Video Projectors

Each 3x5~20 Megapixels Total

MSU 10/20/2003 – 10

Application Barrier Two:Shared Internet Makes Interactive Gigabyte Impossible

NASA Earth Observation System--– Over 100,000 Users Pull Data from Federated Repositories

Two Million Data Products Delivered per Year– 10-50 Mbps (May 2003) Throughput to Campuses

Typically Over Abilene From Goddard, Langley, or EROS– Best FTP with Direct Fiber OC-12: Goddard to U Maryland at 123

Mbps– UCSD to Goddard at 12.4 Mbps– Interactive Megabyte Possible, but Gigabyte is Impossible

Biomedical Informatics Research Network (BIRN) between UCSD and Boston-Similar Story– Lots of Specialized Networking Tuning Used

50-80 Mbps

MSU 10/20/2003 – 11

Solution is to Use Dedicated1-10 Gigabit Lambdas

fc *λ=

(WDM)

Source: Steve Wallach, Chiaro Networks

MSU 10/20/2003 – 12

The UCSD OptIPuter Deployment

½ Mile

SIO

SDSC

CRCA

Phys. Sci -Keck

SOM

JSOE Preuss

6th

College

SDSCAnnex

Node M

Earth Sciences

SDSC

Medicine

Engineering High School

To CENIC

Collocation

Source: Phil Papadopoulos, SDSC; Greg Hidley, Cal-(IT)2

Forged a New LevelOf Campus Collaboration

In Networking Infrastructure

SDSC Annex

JuniperT320

0.320 TbpsBackplaneBandwidth

20X

ChiaroEstara

6.4 TbpsBackplaneBandwidth

2 Miles0.01 ms

Details of UCSD and UIC

Campus InfrastructureTom DeFanti and Phil P

Talks

MSU 10/20/2003 – 13

Chicago Metro-Scale OptIPuter

• Electronic Visualization Laboratory and Northwestern Univ use I-WIRE and OMNInet Fiber to connect clusters

• OMNInet is a 10GE Metro-Scale Testbed (Joe Mambretti)• I-WIRE is State of Illinois Initiative

16x1GigE

UIC

Chicago

MSU 10/20/2003 – 14

Chicago Metro-Scale OptIPuter

Int’l GE, 10GE

Nat’l GE, 10GE

I-WIRE OC-192

16x1 GE 16x1 GE

16x1GE

OMNInet 10GEs

128x128 Calient

64x64GG

16-dual Xeon Cluster at NU

16-dual Xeon Cluster at UIC/EVL

All Processors also Connected by GE to Routers

MSU 10/20/2003 – 15

Studying Both Routed and Switched Lambdas

OptIPuter evaluating both:– Routers

ChiaroJuniperCiscoForce10

– Optical SwitchesCalientGlimmerglass (GG)

UCSD Focusing on IP Routing InitiallyUIC/NU Focusing on Optical Switching initially

MSU 10/20/2003 – 16

Chiaro IP RouterArchitectural Overview

Optical Phased Array (OPA) fabric enables large port count

Global arbitration provides guaranteed performance

Smart line cards NetworkNetwork

Proc.Proc.LineLineCardCard

NetworkNetworkProc.Proc.LineLineCardCard

GlobalGlobalArbitrationArbitration

Optical Electrical

ChiaroOPA

Fabric

NetworkNetworkProc.Proc.LineLineCardCard

NetworkNetworkProc.Proc.Line Line CardCard

MSU 10/20/2003 – 17

InputOptical Fiber

WG #1 WG #128

••••••

UCSD Uses Chiaro Optical Phased ArrayMultiple Parallel Optical Waveguides

OutputOutputFibersFibers

Air Gap

Air Gap

GaAs Waveguides

MSU 10/20/2003 – 18

UIC Calient DiamondWave Switches for StarLight and NetherLight

3D MEMS Micro-Electro-Mechanical Systems) structure128x128 at StarLight64x64 at NetherLight

MSU 10/20/2003 – 19

Year Two Metro-Scale Experimental Network

Fiber link between UCSD and SDSU

MSU 10/20/2003 – 20

Year Two State-Scale Experimental Network

UCSD SDSU

USC UCI

NASAAmes?

Source: CENIC

Identified at UC Irvine: • Cluster Lab• Campus Optical Fiber• CENIC Optical Connections• OptIPuter Up End of 2003

~400 Miles 2 ms

Chiaro

MSU 10/20/2003 – 21

Year Two National-Scale Experimental Network

“National Lambda Rail”Chicago OptIPuter

StarLightNU, UIC

SoCalOptIPuter

USC, UCIUCSD, SDSU

2000 Miles 10 ms

=1000x Campus LatencySource: John Silvester, Dave Reese, Tom West-CENIC

MSU 10/20/2003 – 22

An International-Scale OptIPuter

UKLight

CERN

NorthernLight

MSU 10/20/2003 – 23

OptIPuter Optical Network Architecture

Joe MambrettiInternational Center for Advanced Internet Research

Northwestern University

Oliver YuUniversity of Illinois Chicago

September 2003

MSU 10/20/2003 – 24

OptIPuter Optical Architecture Themes

The OptIPuter Shapes Itself to Meet the Precise Needs of the Application Requirements vs. Today’s Environments in which the Application must be Compromised to Conform to Infrastructure Restrictions The OptIPuter Enables the Creation of Dynamic Virtual Computers Instantiations, Releasing Resources after UseResources Include Optical Networking Components:

– Dynamic Lightpaths Supported by Next Generation Optical NetworksFor the OptIPuter, the “Network” is No Longer a Network, but a Large Scale, Distributed System Bus

– An Optical “Backplane” Based on Dynamically Provisioned Datapaths, Including Lightpaths

Also, the OptIPuter Uniquely Addresses the Needs of Extremely Large Scale Sustained Data Flows – Even Those Exhibiting Dynamic Unpredictable BehaviorsAchieving These Goals Requires New Architecture, Methods and NewTechnologies at All Levels – L1 – L7

MSU 10/20/2003 – 25

Clusters

DynamicallyAllocatedLightpaths

Switch Fabrics

Apps

CONTROL

PLANE

OptIPuter Team Research Agenda

PhysicalMonitoring Multi-Leveled Architecture

MSU 10/20/2003 – 26

Optical Network Architecture Issues

Middleware

Application Protocols

Data Plane Control Plane

Management Plane

Optical Transport

Performance Fault & ConfigurationSecurity

• Traffic Engineering• Topology Info Distribution• Lightpath Selection• Connection Signaling

• Resource Brokering• Advanced Scheduling

MSU 10/20/2003 – 27

Single Domain Model

Control Plane (Signaling Network)

Data Plane (Optical Transport Network)

O-UNI (Signaling)

UNI (Transport)

Lay

er M

anag

emen

t

System Mgt.Management Plane

MSU 10/20/2003 – 28

Multiple Domains Models

UNI (Signaling)

Control Plane

Data Plane

Control Plane

Data Plane

Control Plane

Data Plane

UNI (Transport)

Signaling Gateway Signaling Gateway

MSU 10/20/2003 – 29

Receding Optical Core Cloud User-centric Dynamic Lightpath Provisioning

Optical CoreUser Domain

App.App.

User Domain

App.App.

User Domain

App.App.User Domain

App.App.

App.App.App.App.

Opening Internals of Optical Core

App.App.

App.App.

App.App.

App.App.

MSU 10/20/2003 – 30

OptIPuter Control Plane Paradigm Shift

OptIPuter: Dynamic Services,

Visible & Accessible Resources,Integrated As Required By Apps

Traditional Provider Services:Invisible, Static Resources,

Centralized Management

Invisible Nodes,Elements,

Hierarchical,Centrally Controlled,

Fairly Static

Limited Functionality,Flexibility

Greatly Increased Functionality,Flexibility

MSU 10/20/2003 – 31

Optical Layer Control Plane

Controller

Client Device

Client Controller

ControllerController

Controller

Controller

Client Layer Control Plane

Optical Layer Control Plane

UNI

I-UNI

CI

CICI

Management Plane

Client Layer Traffic Plane

Optical Layer – Switched Traffic Plane

MSU 10/20/2003 – 32

OptIPuter and Inter-Domain Intelligent Signaling

SURFNET/NetherLight(Holland)

CA*net4(Canada) StarLight

IWIRE(Illinois)

ODIN

GMPLS Signaling

OMNInet(Chicago)

PIN

PINPIN

PIN

Gigabit ClusterGigabit Cluster

Source: Oliver Yu PIN (Photonic Inter-domain Negotiator)ODIN (Optical Dynamic Intelligent Signaling)Other Interdomain Projects at CANARIE, U of Amsterdam, et al.

MSU 10/20/2003 – 33

PIN Architecture

Dispatcher

Routing Table

PIN 1

ODIN

Generic signaling message

Translator

Dispatcher

Routing Table

PIN 2

Dispatcher

Routing Table

PIN 3

User BUser A

Inter-domainRouting

GMPLSControl Plane 1

Intra-Domain Signaling

Control Plane 2 Control Plane 3

Intra-Domain Signaling

Domain 1 Domain 2 Domain 3

Inter-domainRouting

Inter-domainSignaling

Inter-domain Signaling

Translator Translator

Source: Oliver Yu

MSU 10/20/2003 – 34

• 8x8x8λ Scalable photonic switch

• Trunk side – 10 G WDM

• OFA on all trunks

OMNInet Testbed Used for OptIPuter Experiments

10 GE10 GE

To Ca*Net 4

StarLight

Photonic Node

DataCom CtrPhotonic

Node

EVL UIC iCAIR, NorthwesternPhotonic

Node10/100/GIGE

10/100/GIGE

10/100/GIGE

10/100/GIGE

10 GEOptera5200

10Gb/sTSPR

Photonic Node

λ4

PP8600

10 GEPP

8600

PP8600

2

3

4

1

λλλ

λ

λλ

λ2

3

1

Optera5200

10Gb/sTSPR

10 GE10 GE

Optera5200

10Gb/sTSPR

2

3

4

1

λλλ

λ

Optera5200

10Gb/sTSPR

2

3

4

1

λλλ

λ

1310 nm 10 GbEWAN PHY interfaces

10 GE10 GE

PP8600

Fiber

KM MI1* 35.3 22.02 10.3 6.43* 12.4 7.74 7.2 4.55 24.1 15.06 24.1 15.07* 24.9 15.58 6.7 4.29 5.3 3.3

NWUEN Link

Span Length

…CAMPUSFIBER (16)

EVL/UICOM5200

LAC/UICOM5200

CAMPUSFIBER (4)

INITIALCONFIG:10 LAMBDA(all GIGE)

StarLightInterconnect

with otherresearchnetworks

10GE LAN PHY (Dec 03)

TECH/NU-EOM5200

CAMPUSFIBER (4)

INITIALCONFIG:10 LAMBDAS(ALL GIGE)

Optera Metro 5200 OFA

NWUEN-1

NWUEN-5

NWUEN-6NWUEN-2

NWUEN-3

NWUEN-4

NWUEN-8 NWUEN-9NWUEN-7

Fiber in useFiber not in use

5200 OFA

5200 OFA

Optera 5200 OFA

5200 OFA

DOT GridClusters

Research Partnership: Nortel, SBC, iCAIR, EVL, ANL

OMNInet is a SONET-Free Zone

MSU 10/20/2003 – 35

OptIPuter and Related Standards Initiatives

P-P InternetworkingIntra-Domain InteroperabilityInter-Domain Interoperability

MPLSGMPLS

Link ManagementCAMP

TE CR LDPTE RSVP

IP-over-OpticalOBGP

IETFInternet Engineering Task Force

iG.ASONUNI

PNNIInter-Domain Interoperability

Intra-Domain Autonomous ProcessTE RSVP

TE CR LDPHierarchical Support

Architecture

ITUInt’l Telecomm Union

O-NNIO-UNI

OIFOptical Internetworking Forum

GHPN-RG

GGFGlobal Grid Forum

White Paper on Optical GridReleased During GGF9 in Oct