OptIPuter Physical Testbed at UCSD, Extensions Beyond the Campus Border

Philip Papadopoulos and Cast of Real Workers:Greg HidleyAaron Chin

Sean O’ConnellMax Okumoto

Praveen KumarMason Katz

David Hutches

Physical Campus Connections

• O(300) nodes (Storage, Compute, Visualization)• O(30) switches• How nodes are connected to the network

– Optical Core– Site switches (2 x 10GigE) – 48 port GigE– Rack Switches

• 24 or 48 port GigE copper

UCSD Packet Test Bed – Year 2/3

Quartzite Extensions

• Funded as NSF Major Research Instrumentation Award – Companion to OptIPuter

• Observation: Packet-only switch structure not rich enough for OptIPuter research

• 300+ OptIPuter Nodes– Viz (3 tiled displays), Storage (48 node, 300 spindles), Compute several

hundred (x86, Opteron, Itanium never materialized)• Current UCSD OptIPuter network is channel-bonded GigE. Starting

some 10 GigE Deployment (network not “fat” enough for optIPuter research)– Very odd behavior with 10-year old etherchannel technology

• Quartzite more closely matches the network capability to the nodes.– At the end of three years: 0.5 Terabits into the Quartzite Switching Core

• Quartzite fundamental capability: Build hybrid networks– Packet-switched, Circuit-switched, Wavelength-switched. Hybrid

combinations, and reconfigurable

Building Blocks of Quartzite

• CWDM is inexpensive ($2500/connected endpoint)– Colored GBIC

• Use 8 GigE frequencies centered at 1550nm (+/- j*20nn (j=1,2,3,4))• + LR 10 GigE at 1320nm. Colored 10GigE XenPAK or SFP may be available by end of

proposal

– Passive optical multiplexers (Coarse). 9 channel, 1550nm centered GBICs + 1320nm Passband for 10GigE

– Comment: Since proposal (January 2004), uncooled DWDM lasers are more readily available.

• OOO optical switch (commercial) Glimmerglass. Use as optical patch panel• Custom-built wavelength-selective switch from Lucent• Standard packet switch-router. Want to extend the Chiaro but $$ limit this

expansion– 32-port (10GigE) capable 6509 switch ordered with 8 10-GigE ports

• Existing single mode fiber plant (already tested and terminated)– 4 pairs/site on the UCSD campus

Quartzite Year 1 Deployment Plans

• Deploy OOO Switch– Optical “patch

panel”

• Add some L2 10 GigE connections

• Initial Experience with CWDM

Year 2 Deployment Plans

• Add Several CWDM Channels

• Deploy Wavelength Switch

• Begin Significant 10 GigE Buildout– Packet Switch

– 10 GigE into some endpoint nodes

Final Planned Configuration

• >50 endpoints connected at 10 GigE

• >= 32 Packet switched

• >= 32 switched wavelengths

• >= 300 Connected endpoint

Glimmerglass Switch – Now Deployed

A Whole Host of Real Issues

• Network:– Where are nodes connected?– How are switches interconnected

• Channel bond? 10-GigE?• Are there wavelengths to be allocated? How?• Can I build a VLAN that traverses specific physical links• Can have a library of different network configs for the testbed

• Node Software:– Can I have root on nodes? How do I specify a particular OS Image on a

node?– What nodes can I have? For how long? How do I determine that I am

(or not) in competition for resources on the testbed• Grid Software:

– What Grid software is available as base configuration? What should be• Performance

– Why does my long-distance channel give me asymmetric performance

Network Inventory Management

• Schema designed to support management of both physical and virtual topologies. Physical topology is the network with connections as it is. Virtual topology shows the actual network flow paths.

• Physical Topology Schema components– Switch model ( Mfr, Model name)– Switch unique information ( MAC, IP, Hostname etc )– Host model ( Processor, Memory etc )– Hosts unique information ( MAC, IP, Hostname etc)– Connections

( Physical connection among switch or host entities )

– Trunk based connections ( Ports participating in an physical trunk )• Connections are the basic primitives which allow

us to build, visualize and navigate the topology. Any information corresponding to the network can be had by an join of connections and other tables in the schema.

PHYSICAL TOPOLOGY GRAPH

• Logical topology adds an VLAN table to the physical topology tables.

– VLAN composed of trunks. – Each Trunk can be a single/multiple port to

port connection between same set of switches– Schema supports retaining VLAN id when

modifying trunks and vice-versa.

Optiputer Network Inventory Management

• Schema contains network information needed to program the network to construct virtual topologies. VLAN table in conjunction with Connections and Trunks can be used to create the needed setup. User can specify inputs in terms of IP address or Host names, which will be mapped to the tables above through host and switch tables.

• Database implemented in MySQL with Python wrappers to insert data and generate output using GraphViz. • Work in progress to automate data collection.

LOGICAL TOPOLOGY (Single VLAN) GRAPH

OS and Software Integrations

• Rocks configurations on endpoints allow us to build libraries of OS configs.

• Rolls allow programmatic extensions to a complete cluster installation.– E.g., Grid, Scheduler, Kernel Rolls allow overwrite/extension– Visualization Roll captures tiled-display wall software

Extensions to Wider Area

Southern California CalREN-XD Build Out

UCSD

StarLight Chicago

UIC EVL

NU

Expanding the OptIPuter LambdaGrid

PNWGP Seattle

NLR

NLR

NASA Goddard

Level 3

NASA Ames

NASA JPL

NetherLight Amsterdam

U Amsterdam

ISI

UCI

CENIC San Diego GigaPOP

CENIC Los Angeles

GigaPOP

CalREN-XD

22

8

8

NLRNLR2

SDSU

CICESE

via CUDI

CENIC/Abilene Shared Network

1 GE Lambda

10 GE Lambda

Southern California CalREN-HPR