canarie ca*net 4 design document last revised april 22 2001 version 1.20 obgp documentation and...
TRANSCRIPT
CANARIE http://www.canarie.ca
CA*net 4 Design DocumentLast Revised April 22 2001
Version 1.20OBGP documentation and latest version of this
document can be found at
http://www.canet3.net
[email protected]: +1.613.785.0426
The Concept for CA*net 4 Conventional optical networks are built on the paradigm that a central entity
has control and management of the wavelengths It therefore must have control of the edge device for the setup and tear down of
the wavelengths Will central control and management scale to millions of edge device and
thousands of optical wavelengths? Customer empowered optical networks are built on the paradigm that
customer owns and controls the wavelengths (Virtual Dark Fiber) Customer controls the setup, tear down and routing of the wavelength between
itself and other customers Customer may trade and swap wavelengths with other like minded customers
ultimately leading to wavelengths as market commodity How do you design a network architecture if the routing and control of
wavelengths is under the control of the customer at the edge? Network is now an asset, rather than a service
Analogy to time sharing computing in the early 1970s versus customer owned computers or client-server computing
Condo Fiber & Wavelengths Condo fiber means that separate organizations own individual strands of
fiber in a fiber cable Each strand owner responsible for lighting up the strand Collectively responsible for sharing costs of maintenance on fiber cable,
relocation etc Condo wavelengths
Number of parties share in the cost of a single strand and that light up with an agreed upon number of wavelengths
Wavelengths are portioned based on percentage ownership With condo fiber and condo wavelengths institutions can treat network as an
asset just like purchasing a computer, rather than a service as today
Research Network Issues Research and Education networks must be at forefront of new network
architecture and technologies Should be undertaking network technology development that is well ahead
of any commercial interest But any network architecture can only be validated by connecting real users
with real applications and must solve real world problems Test networks per se are not sufficient
There is a growing trend for many schools, universities and businesses to control and manage their own dark fiber
Can we extend this concept so that they can also own and manage their own wavelengths?
Will “empowering” customers to control and manage their own networks result in new applications and services similar to how the PC empowered users to develop new computing applications?
CA*net 4 Research Objective To deploy a network architecture where the GigaPOPs and institutions at the
edge manage and control their own fiber and their own wavelengths Condominium fiber and condominium wavelengths
To deploy a novel new optical network of distributed optical IXs that gives GigaPOPs and communities at the edge of the network (and ultimately their participating institutions) the ability to setup and manage their own wavelengths across the network and thus allow direct peering between GigaPOPs on dedicated wavelengths and optical cross connects that they control and manage
To allow the establishment of wavelengths by the GigaPOPs and their participating institutions in support of eScience and grid applications to support true peer-to-peer networking
To allow connected regional and community networks to setup peering relationships with CA*net 4 for collaborative research and education and eScience applications
To partner with private sector in building “carrier neutral” distributed optical Internet exchange facilities across Canada and developing new services in fungible wavelengths to enable customer empowered optical networks
Current View of Optical Internets
Big Carrier Optical Cloud using MPS or ASON for management of wavelengths
for provisioning, restoral and protection
Carrier controls and manages edge devices
Optical VLAN
Customer
ISP
AS 1
AS 2
AS 3
AS 1AS 4
AS 5
UNI
NNI
Customer Empowered Metro Network
Carrier Neutral IX &OBGP switch
City A
City B
City C
Carrier Neutral IX& OBGP switch
Condo Dark Fiber
Condo Wavelengths
Condo Wavelengths
OBGP switch
OBGP switch
Future Optical Networks
Customer A
Customer B
Customer C
Customer D
Customr A elects to cross connect with Customer C rather than D
Massive peering at the edge
Condo Fiber
Condo Wavelength
CA*net 4 Research Areas New optical technologies that support customer empower networking
OBGP, CWDM, hybrid optics and HWDM, customer controlled optical switches
BGP scaling issues Object Oriented Networking
Wavelengths and optical switch treated as an object and method to be incorporate into middleware
Or treated as fungible product Distributed Computing Applications and Grids
Wavelength Disk Drives (WDD) eScience Grids for weather forecasting, forestry management, education, health,
etc
Object Oriented Networking Combines concepts of Active Networks and Grids
See DARPA See Globus
Customer owns sets of wavelengths and cross connects on an optical switch Network elements can be treated as a set of objects in software
applications or grids Complete with inheritances and classes, etc Rather than distributed network objects ( e.g. Java or Corba) distributed
object networks In future researchers will purchase networks just like super computers,
telescopes or other big science equipment Networks will be an asset – not a service Will be able to trade swap and sell wavelengths and optical cross connects
on commodity markets
Advantages of OON With massive peerings to the edge, the loss of one peer is not catastrophic
No need for restoral or protection paths or ring architectures Networks look more like “star bursts” rather than “ring of rings”
See C Labovitz ACM Sigcomm Aug 2000 – massive peering helps faster convergence
May solve problem of scaling large networks Today M carriers building meshed networks to N customers with
resultant M*N2 requirement for wavelengths With OON the global requirement for wavelengths grows at X*N
where X= average number of wavelengths per customer
Example OON Earthquake Visualization Grid
Globus Middleware Begin Establish connection to other grid participants
Network Object – wavelength to STAR LIGHT – Chicago Network Object – wavelength to Research center Amsterdam Network Object – wavelength to SDSC Visualization Computer Network Object – wavelength to Seismology Center Calgary Link objects and create grid
Run Visualization Release Network objects
Globus Middleware End Earthquake Visulization End
Napster OON
University in Canada willing to exchange these wavelengths
Montreal-NY blue NY-Amsterdam red Chicago-Montreal green
Wants these wavelengths Chicago-SDSC - purple SDSC- Hawaii - red Chicago- Chile - yellow
University in Chicago willing to swap these wavelengths
Chicago to SDSC – purple SDSC- Hawaii – Red Chicago to Argonne - Blue
Wants wavelength to these locations
Chicago to Toronto - yellow
“National grand challenge" e-research projects are on the horizon: with the next generation network, interconnecting to school and community networks, Canadian researchers could use the thousands of computers in schools and communities distributed across Canada
Students at schools and ultimately members of the public could be full participants in basic reserach
The next generation research network should be designed to encourage and enable projects such as these
The eScience Vision
Wavelength Disk Drives
Vancouver
Computer data continuously circulates around the WDD
Calgary
Regina
Winnipeg
Ottawa
Montreal
Toronto
Halifax
St. John’s
Fredericton
Charlottetown
CA*net 3/4
WDD Node
eScience Grid
Customer A
Customer B
Customer C
Customer DWDD Grid
Customers autonomously create WDD ring for high
performance applications
Wavelength Disk Drives
CA*net 4 will be “nation wide” virtual disk drive for grid applications
Big challenges with grids or distributed computers is performance of sending data over the Internet TCP performance problems Congestion
Rather than networks being used for “communications” they will be a temporary storage device
Ideal for “processor stealing” applications
OBGP Proposed new protocol to support control and management of wavelengths
and optical switch ports Control of optical routing and switches across an optical cloud is by the
customer – not the carrier – true peer to peer optical networking Use establishment of BGP neighbors or peers at network configuration
stage for process to establish light path cross connects Customers control of portions of OXC which becomes part of their AS Optical cross connects look like BGP speaking peers – serves as a
proxy for link connection, loopback address, etc Traditional BGP gives no indication of route congestion or QoS, but with
DWDM wave lengths edge router will have a simple QoS path of guaranteed bandwidth
Wavelengths will become new instrument for settlement and exchange eventually leading to futures market in wavelengths
May allow smaller ISPs and R&E networks to route around large ISPs that dominate the Internet by massive direct peerings with like minded networks
Opportunity for carrier and industry partners
To participate in a novel new Internet architecture that will allow customers to manage and control their own wavelengths anywhere across the network
Very attractive technology for Tier 2 ISP, research networks and ASPs Yahoo and Cable and Wireless have already started down this path It will allow them to create their own network topologies
To provide a valuable new service for customers that will allow them to reduce Internet transit costs by as much as 75%
To develop new value added services in IX brokering and management To develop new fungible trading services in bandwidth trading and brokering To experiment with new long haul optical technologies that will dramatically
reduce cost of long haul transmission
CA*net 4 Possible Architecture
Vancouver
Calgary ReginaWinnipeg
Ottawa
Montreal
Toronto
Halifax
St. John’s
Fredericton
Charlottetown
Chicago
Seattle
New York
Europe
Customer controlledoptical switches
Layer 3 aggregation serviceOptional Service Available to any GigaPOP
Large channel WDM system
Wavelength Scenarios
Vancouver
Calgary
ReginaWinnipeg
Toronto
Halifax
St. John’s
Seattle
Montreal
Workstation to Workstation Wavelength
University to University Wavelength
CWDM
BCnet
RISQ
GigaPOP to GigaPOP WavelengthCampus OBGP switch
Wavelength Setup
AS 1
AS 2
AS 3
AS 4
AS 5
AS 6
Dark Fiber
Wavelength Object owned by primary customer Wavelength Subcontracted by primary customer to a third party
AS 1- AS 6 Peer
AS 2- AS 5 Peer
2
3
4
5
6
17
8
9
1012
13
14
15
Regional Network
Regional NetworkUniversity
University
ISP router
Wavelength Logical Mapping
AS 1
AS 2
AS 3
AS 4
AS 5
AS 6
Primary Route
Backup Route
AS 1- AS 6 Peer
AS 2- AS 5 Peer
2
3
4
5
6
17
8
9
1012
13
14
15
Regional Network
Regional NetworkUniversity
University
ISP router
Resultant Network Topologies
AS 1
AS 5
AS 6
13
14
15
Regional Network
Regional NetworkUniversity
University
AS 2
2
1
7
ISP router
8
9
5
9
1
2
8
6
7
310
5
12
10OBGP
Potential OBGP Peering
BGP Peering on switches at the edgePacket Forwarding in the core
Possible CA*net 4 Node
4 Channel GbE CWDM to local GigaPOP
Carrier A
Carrier B
OC48 DWDM
OC192 DWDM
8 Channel GbE CWDM to next CA*net 4 node
2xGbE
10xGbE
Optional Aggregating Router
Carrier A OBGP Switch
CA*net 4 switch
Physical Wavelength Configs
OC192 DWDMCarrier B
10xGbE 10xGbE
Carrier A
OC48 DWDM
2xGbE 2xGbE
OBGP links
ORAN B ORAN A ORAN C ORAN D
Dark fiber +CWDM
2xGbE
1
23 4
CA*net 4
5
6
7 8
Logical Wavelength Configs
OBGP links
ORAN B
ORAN A
ORAN C ORAN D
Carrier A
Carrier B
GbE over CWDM
CA*net 4
GbE over 10GbE over OC-192 DWDM
GbE over 2GbE over OC-48 DWDM
1 2 3 4
5
Possible Wavelength Assignment Illustrative purposes only Assume 150 wavelength system across Canada 50 wavelengths assigned to provincial networks based on a number of
criteria including ability to extend wavelengths into provincial network and requirements for high bandwidth applications
ORANs encouraged to extend wavelengths to individual institutions Institutions encouraged to deploy optical switches
On all cross sections a minimum of 100 wavelengths dedicated to CA*net 4 and carrier partners
2 wavelengths dedicated to CA*net 4 layer 3 aggregation service (looks like old CA*net 3)
10 wavelengths (and OCX ports) reserved for temporary applications like Grids or eScience
A wavelength and OXC port bartering and exchange mechanism so that ORANs can swap wavelengths will be an important requirement
Example Physical Architecture CANARIE builds heterogeneous network made from many sources e.g (illustrative
purposes only): dark fiber from St. John to Halifax using ULH 16 channel POS dark fiber condo from Halifax to Fredericton using 16 channel 10GbE Condo wavelengths from RISQ from Edmonston to Ottawa sharing 32 channel 10GbE
system dark fiber from Ottawa to Winnipeg with Onet using 16 channel POS at OC-192 Condo wavelengths from Bell Canada from Montreal to Chicago as part of a 140
wavelength system wavelengths from Telus from Chicago to Winnipeg as part of a 140 wavelength system dark fiber from GT Telecom from Winnipeg to Calgary using 16 channel 10GbE wavelengths from Shaw from Calgary to Vancouver as part of a 32 channel 10GbE
systems wavelengths from 360 Networks from Halifax to London as part of a 400 wavelength
system Wavelengths from Teleglobe from Seattle to Honolulu – Sydney – Tokyo - Seoul
OBGP Variations1. OBGP Cut Thru
OBGP router controls the switch ports in order to establishes an optical cut through path in response to an external request from another router or to carry out local optimization in order to move high traffic flows to the OXC
2. OBGP Optical Peering External router controls one or more switch ports so that it can establish direct
light path connections with other devices in support peering etc
3. OBGP Optical Transit or QoS To support end to end setup and tear down of optical wavelengths in support of
QoS applications or peer to peer network applications
4. OBGP Large Scale To prototype the technology and management issues of scaling large Internet
networks where the network cloud is broken into customer empowered BGP regions and treated as independent customers
OBGP Optical Peering Primary intent is to automate BGP peering process and patch panel process Operator initiates process by click and point to potential peer Original St. Arnaud concept
Uses only option field in OPEN messages Requires initial BGP OPEN message for discovery of OBGP neighbors Virtual BGP routers are established for every OXC and new peering relationships are
established with new BGP OPEN message Full routing tables are not required for each virtual router No changes to UPDATE messages No optical transit as all wavelengths are owned by peer Uses ARP proxy for routers on different subnets
Wade Hong Objects concept Uses an external box (or process) to setup optical cross connects SSH is used to query source router of AS path to destination router Each optical cross connect is treated as an object with names given by AS path Recursive queries are made to objects to discover optical path, reserve and setup NEXT_HOP at source router is modified through SSH End result is a direct peer and intermediate ASs disappear Requires all devices to be on same subnet
OBGP Optical Transit Wavelengths are under control of another entity who has temporarily allowed
them to be available for transit Viagenie – Marc Blanchet and Florent Parent
Designed specifically for optical transit applications Uses MBGP and establishes new address family for OBGP Community tags are used to advertise availability of optical paths as part
of NLRI and COMMUNITY TAG Reservation and setup is done by advertising update NLRI message Exploring using CR-LDP & RSVP-TE with AS loose routing for path
reservation and setup Changcheng Huang
The same NLRI message is sent back and forth and modified to indicate first availability of wavelengths, reservation and setup
Over rides loop back detection in RIBS for advertised NLRI messages
Target Market for OBGP University research and community networks who are deploying
condominium fiber networks who want to exchange traffic between members of the community but who want to maintain customer control of the network at the edge and avoid recreating the need for aggregating traffic via traditional mechanisms
E.g. Ottawa fiber build, Peel County, I-wire, SURAnet, G-Wire, CENIC DCP, SURFnet, etc etc
Next generation fiber companies who are building condominium fiber networks for communities and school boards and who want to offer value added fiber services but not traditional telcommunications service
E.g. C2C, Universe2u, PF.net, Williams, QuebecTel, Videotron, etc Next generation collocation facilities to offer no-cost peering and wavelength
routing Metromedia, Equinix, LINX, PF.net, LayerOne, Westin, PAIX, Above.com,
Colo.com, etc etc Over 500 Ixs and carrier hotels worldwide
OBGP Peering
Possible technique for allowing automatic peering at IXs between consenting ISPs
External routers are given control of specific ports on the OXC The router that controls switch can act as an optical route server notifying all
peers of any new consenting OBGP peers External routers signal to each other if they wish to setup direct optical
connection Choice of partner can be based on size of traffic flows Partners can be changed through a routing flap
Only see each other’s customers routes – not the default core
OIX using OBGP
AS 100160.10.10.0
AS 200170.10.10.0
AS 300180.10.10.0
AS 400190.10.10.0
Institution A
Institution B
Institution C
Institution D
Figure 10.0
Switch Ports are part of institution’s AS
Transport Architecture Heterogeneous transport architectures used on backbone links Type of transport architecture on each link determined by length of link between O-
ADMs, GbE-ADMs or OBGP switches, requirement for optical repeaters or regenerators, etc
Examples: 8 or 16 channel GbE used on short haul links (up to 2000 km) between OADMs or
OBGPS; or OC-192 Ethernet over SONET with multiplexed 10 single GbE or trunked 10GbE;
or Proprietary 8 channel 2 x GbE multiplexed into OC48 optics with FEC wrapper
Repeaters: GbE or 10GbE 2R transceivers every 50-80 km combined with GbE or 10Gbe 3R
switches every 200 – 400 km; or Traditional EDFAs at 1550nm every 50- 80 km with OC-192 regenerators at every
200-400km; or All optical broadband: Counter rotating Raman amplifiers, multi band EDFAs,
EFFs, dispersion correction fiber, etc
Tributary Architecture
Customer can connect through OADM,Gbe-ADM, direct to OBGP switch or through CA*net 4 router
Customer access link is either GbE or trunked 10GbE (I.e. 10 separate GbE channels
In future customer will have a choice of protocols, but for now GbE will be basic standard across the network
Switch Architecture Low speed MEMs or similar capacity switch
Could also use non blocking GbE switch Switch can also be distributed across an optical network using GMPLS or
ODSI Each switch component can be controlled by a socket/port by any external
network element with appropriate security mechanisms If OXC used for traffic engineering or QoS then controlling router manipulates
both input and output ports If OXC used for distributed peering then participating AS only owns either
INPUT or OUTPUT ports Eventually switches can also support optical trunking of many optical paths Switch commands are kept very simple, leaving all complexity to OBGP
messages Switch does not know or care the direction of the wavelength – that is
established with OBGP protocol