CANARIECA*net 4 Planning

http://www.canarie.ca http://www.canet3.net

Bill.St.Arnaud@canarie.caTel: +1.613.785.0426

Customer Empowered Network

Carrier Neutral IX

City A

City B

City C

Carrier Neutral IX

Condo Dark Fiber

Condo Wavelengths

Condo Wavelengths

What is eScience? The ultimate goal of e-science is to allow students and eventually members

of the general public to be full participants in basic research. Using advanced high speed networks like CA*net 4 and novel new concepts

in distributed peer to peer computing, called “Grids” many research experiments that used to require high end super computers can now use the computer capabilities of thousands of PCs located at our schools and in our homes.

High performance computers that are part of C3.ca can be seamlessly integrated with eScience distributed computers using CANARIE Wavelength Disk Drive over CA*net 4

Allows researcher access to the significant computational capabilities of all these distributed computers at our schools and homes

With e-science it might be possible that the next big scientific discovery could be by a student at your local school.

ALTA Cosmic Ray eScience The earth is constantly

bombarded by subatomic particles from space, with an energy spectrum that reaches far higher than any terrestrial accelerator could hope to probe. 

At the highest energies such showers can be detected at the Earth’s surface over areas on the order of 100 square kilometers. 

It is believe some of these cosmic rays were created at the creation of the universe

Will allow researchers to gainer a deeper understanding of deepest reaches of space and time 

ALTA Cosmic Ray eScience The ALTA project is a collaborative scientific research project

involving the University of Alberta Center for Subatomic Research and over 50 high schools across Canada in the area of cosmic ray detection.

Teachers and students actively contribute to the physics research while learning about an exciting area of modern science. 

Distributed computing at schools will be required to analysize data from sensors in near real time

Program has now expanded into USA and soon countries around the world

CHICOS (California HIgh school Cosmic ray ObServatory), Caltech, UC/Irvine and Cal State/Northridge, California, USA.

CROP (the Cosmic Ray Observatory Project), University of Nebraska, Lincoln, NE, USA.

WALTA (WAshington Large area Time coincidence Array), University of Washington, Seattle, WA, USA.

SALTA Roaring Fork Valley area of Colorado

Neptune – Undersea Grid

Wavelength Disk Drives CA*net 4 will be “nation wide” virtual disk drive for grid

applications Big challenges with grids or distributed computers is

performance of sending data over the Internet TCP performance problems Congestion

Rather than networks being used for “communications” they will be a temporary storage device

Ideal for “processor stealing” transaction intensive applications where you don’t know where the next available processor is located

CFD Visualization

Wavelength Disk Drives

Vancouver

Computer data continuously circulates around the WDD

Calgary

Regina

Winnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

Fredericton

Charlottetown

CA*net 3/4

WDD Node

WDD Architecture

Vancouver

Calgary Halifax

WDD Node

WDD Partners:CANARIE, Can-Sol, Viagenie

CRC, Carleton U, MACIC3.Ca, Memorial, DalhousieUdeMontreal, UoToronto,

SFU, UoAlberta, BCnet

MemorialDalhousieUdeMontrealUoTorontoUoAlberta

SFU

WDD Node

Forest FireModeling

Raster Engine

WDD Node

CRC

1. Forest Fire Modeling Raster Engine injects 64K x 64K raster computational tasks into WDD ring

2. Tasks circulate in WDD ring and first available SGI processor removes next task out of the ring and completes computation

3. The SGI writes back the task onto the ring where it is received by Forest Fire Raster Engine and results displayed on X-Window terminal at CRC

WDD Ring on CA*net 3

Forest Fire Modeling eScience Emergency officials and civic

defense officials need to model forest fires in real time

But each forest fire model may take hours to compute

By utilizing thousands of distributed computers at our schools and Wavelength Disk Drive on CA*net 4 network forest fire models in near real time

First prototype to be demonstrated on CA*net 3 in May using 256 SGI processors across the country on WDD

CA*net 4 Possible Architecture

Vancouver

Calgary ReginaWinnipeg

Ottawa

Montreal

Toronto

Halifax

St. John’s

Fredericton

Charlottetown

Chicago

Seattle

New York

Europe

Customer controlledoptical switches

Layer 3 aggregation serviceOptional Service Available to any GigaPOP

Large channel WDM system

STAR LIGHT Interconnection? We see STAR LIGHT, CA*net 4, DTF and Vancouver Transit exchange facing

same design issues How do we signal interconnect wavelengths (SDH/SONET subchannels) between

STAR LIGHT participants? Like STAR TAP we will probably need a mix of Layer 1-3 solutions

Layer 1 cross connect ATM plus Layer 3 router and/or route server

Current ATM approaches Full mesh ATM like current STAR TAP

Not possible with wavelengths or SDH/SONET channels PVC created on demand

E.g Peer maker at MAEs

STAR LIGHT Options Layer 0 - Patch panel or optical switch

Needs common wavelength and protocol Not easily subject to change and will not allow multiple peers

Layer 1 - SDH/SONET cross connect switch Issues related to how identify and address SDH/SONET channels

Layer 2 - GMPLS using IP and SONET/optical switch Main thrust of industry –see Juniper/Nortel, Accelight, Cisco, NTT Requires significant centralized management

Layer 2 -Map SDH/SONET channels to GbE channels & use GbE switch Layer 3 - Each network terminates on its own router & routers meshed together

N squared meshing Layer 3 - BIG ROUTER

Will it scale and needs central management and AS Layer 4 – OBGP with CWDM with optical switch

Each CWDM wavelength mapped to SDH/SONET channel Control of switch is by research networks

OBGP Status Report OBGP first draft submitted to IETF Prototype working at Carleton U We want input on next steps for OBGP and see if it will fit within

STAR LIGHT plans Key features:

SDH/SONET & Optical cross connects controlled by attached networks SDH/SONET & Optical cross connects identified by IP addresses & AS RPSL with OON extensions is database used to query who is connected

at switch and at what port BGP OPEN message is used like Peer maker to request optical peering

across the switch BGP UPDATE message and community Tags ( and maybe GMPLS)

will be used to setup multihop wavelengths

OBGP Proposed new protocol to support control and management of wavelengths

and optical switch ports Control of optical routing and switches across an optical cloud is by the

customer – not the carrier – true peer to peer optical networking Use establishment of BGP neighbors or peers at network configuration

stage for process to establish light path cross connects Customers control of portions of OXC which becomes part of their AS Optical cross connects look like BGP speaking peers – serves as a

proxy for link connection, loopback address, etc Traditional BGP gives no indication of route congestion or QoS, but with

DWDM wave lengths edge router will have a simple QoS path of guaranteed bandwidth

Wavelengths will become new instrument for settlement and exchange eventually leading to futures market in wavelengths

May allow smaller ISPs and R&E networks to route around large ISPs that dominate the Internet by massive direct peerings with like minded networks

Wavelength Scenarios

Vancouver

Calgary

ReginaWinnipeg

Toronto

Halifax

St. John’s

Seattle

Montreal

Workstation to Workstation Wavelength

University to University Wavelength

CWDM

BCnet

RISQ

GigaPOP to GigaPOP WavelengthCampus OBGP switch

Wavelength Setup

AS 1

AS 2

AS 3

AS 4

AS 5

AS 6

Dark Fiber

Wavelength Object owned by primary customer Wavelength Subcontracted by primary customer to a third party

AS 1- AS 6 Peer

AS 2- AS 5 Peer

2

3

4

5

6

17

8

9

1012

13

14

15

Regional Network

Regional NetworkUniversity

University

ISP router

Wavelength Logical Mapping

AS 1

AS 2

AS 3

AS 4

AS 5

AS 6

Primary Route

Backup Route

AS 1- AS 6 Peer

AS 2- AS 5 Peer

2

3

4

5

6

17

8

9

1012

13

14

15

Regional Network

Regional NetworkUniversity

University

ISP router

Resultant Network Topologies

AS 1

AS 5

AS 6

13

14

15

Regional Network

Regional NetworkUniversity

University

AS 2

2

1

7

ISP router

8

9

5

9

1

2

8

6

7

310

5

12

10OBGP

Potential OBGP Peering

BGP Peering on switches at the edgePacket Forwarding in the core

OBGP Variations1. OBGP Cut Thru

OBGP router controls the switch ports in order to establishes an optical cut through path in response to an external request from another router or to carry out local optimization in order to move high traffic flows to the OXC

2. OBGP Optical Peering External router controls one or more switch ports so that it can establish direct

light path connections with other devices in support peering etc

3. OBGP Optical Transit or QoS To support end to end setup and tear down of optical wavelengths in support of

QoS applications or peer to peer network applications

4. OBGP Large Scale To prototype the technology and management issues of scaling large Internet

networks where the network cloud is broken into customer empowered BGP regions and treated as independent customers

OBGP Optical Peering Primary intent is to automate BGP peering process and patch panel process Operator initiates process by click and point to potential peer Original St. Arnaud concept

Uses only option field in OPEN messages Requires initial BGP OPEN message for discovery of OBGP neighbors Virtual BGP routers are established for every OXC and new peering relationships are

established with new BGP OPEN message Full routing tables are not required for each virtual router No changes to UPDATE messages No optical transit as all wavelengths are owned by peer Uses ARP proxy for routers on different subnets

Wade Hong Objects concept Uses an external box (or process) to setup optical cross connects SSH is used to query source router of AS path to destination router Each optical cross connect is treated as an object with names given by AS path Recursive queries are made to objects to discover optical path, reserve and setup NEXT_HOP at source router is modified through SSH End result is a direct peer and intermediate ASs disappear Requires all devices to be on same subnet