national backbone extension: rolling out multisite gigabit connections over fibre optics

National Backbone Extension:Rolling out Multisite Gigabit connections over

Fibre Optics

Brian Boyle & Mike Norris

Topics1. NBE – in context2. Aims of the project

3. What has been achieved

4. Under the bonnet

5. The bigger picture

6. Summary

7. Next steps

8. Discussion2

NBE – in context• Building on the NBN• New requirements – hybrid networking• New paradigm of service and support• Proposal and resources• Support from stakeholders and clients

3

Bandwidth

BA

C

Users

2Mbit/s Gigabit

Bandwidth Use

HPC facility

ICHEC traffic

ICHEC latency

General Internet

Daily traffic

Yearly traffic

NBE – the project• Project plan and management• Inventory of sites• Criteria, priorities• Surveys and schedules• Procurement, multivendor scenario

7

Project organisationJohn Boland CEO Project sponsor

Steering group Victor Reijs Mike NorrisJohn Boland

Eoin KennyProject Manger

WG4Network

management/provisionCormac/Daniel

Out sourcing company Brian N

Out of Band

WG5Config tool

Cillian Sharkey

WG3Network Design

Eoin KennyDaniel LeteOwen Byrne

WG6 PoP Planning

StephenColm

WG2Fibre procurement

Owen ByrneLouis TwomeyPaul Cosgrove

WG1Client LiasonMike NorrisBrian Boyle

WG0Project management

Eoin Kenny

WG7DWDM Network

SiemensEoin

Project goals

• A network for connecting clients, with the following features

• High bandwidth• Multi-service• Resilient• Scaleable• Flexible• Secure

9

Goals – Multi-service

– Service orientated connectivity, e.g:• Private secure connection per Institute to An Chéim• Interconnectivity with the Government VPN

– Not just focused on internet reachability– Capable of delivering access to multiple services using

the same infrastructure– Advanced networking protocols supported – IPv6,

multicast

10

Goals - Resilience

• Seek to build in circuit, equipment, path, and routing diversity

• Equivalent service levels on primary and secondary• Using multiple ducts, termination points, equipment• Using a robust routing technology (BGP) • Can make use of existing connectivity via other service

providers

11

Goals - Scalability• Can deliver multiple 1 Gigabit/s circuits• Can operate at line speed end to end• Capable of offering even higher speeds subject to

requirements (10 Gbps, higher)• 10 Gbit/s access for clients being explored• Technology refresh available for 40 Gbit/s…• …and similar for 100 Gbps in development

12

Goals - Flexibility• Not subject to telco business case scenarios• Not tied by contract to any particular network

implementation technology or model• Can use a mix of fibre and traditional managed Ethernet

circuits• Fully compatible with existing networks (no big bang

changes)• Capable of delivering wavelengths to clients if the need is

proven

13

Client perspective• Client priorities are key throughout• Aim is to keep change to client sites minimal• HEAnet installs managed routers and switches in

addition to current network equipment• When ready, services can be tested and migrated

individually• No Big Bang change• No change without explicit client agreement

14

Typical installation• Research of available providers (fibre, managed circuits),

usually via Tender process• Client site survey by provider with client

Paths identified, leading to “way leave” contracts• Civil works commence• Fibres are pulled into building, and subsequently spliced• Routers and switches are delivered, configured by HEAnet• Services can be tested and used by client

15

New link services• Point-to-point Ethernet links (p2p)

– 1 Gbit/s, Ethernet framed– Ethernet Port Service– Ethernet VLAN Service– No aggregation– Automatic monitoring/graphing

16

Ethernet Port and VLAN services

Some examples• Services

- connectivity- institutional access and security- p2p

• Applications- Oireachtas- ICHEC- An Chéim- UCLP

• Client examples- DCU

18

LAN Extensions – Ryan Academy• Centralise services, both voice and data• Servers centrally located at DCU• IP Telephony Extension to Nortel Meridian• Seamless telephony integration• Security cameras and DVR remote view and monitoring• Buildings Management System• High-End Video Conferencing > 2 Meg/sec• Reliability• No onsite technical support ( low maintenance )

19

LAN Extensions – Mater Dei Institute

• Sister College of DCU• All Servers consolidated in DCU• Previous difficulties with server location in MD

( Dust and no Aircon )• Minimal on-site technical support

( mainly helpdesk 2/3 days per week )• No requirement for Sysadmin• Reliability of service

20

Under the bonnet• DWDM, CWDM• Ethernet switches• MPLS• New routing strategies• Out of band access for network management• CPE, routers• Outsourcing of L2 ops management• Provisioning tool• Contributions, SLA

21

Multiple sources

• We make use of many fibre providers to connect at the lowest cost for our requirements– E-net– Dublin City Council– ESBT– BT Ireland– NTL– Aurora

22

Fibre network end 2006 – approx 2300km of fibre

MAN design13/01/06

National Backbone(10G)

Core Ring(10G)

1490nm

Galway PoP

Clinical Science(UCHG)

1490nm 1530nm 1610nm

1610nm1590nm1570nm

1570nm

1550nm1530nm1510nm

1510nm

1470nm

1470nm

mux/demux

mux/demux

Templates

Galway Regional λ Planning { } everything between these will form part of the future MAN ring. Currently only a point-to-point between NUIG and the Clinical Science building exists. This will later form part of the proposed ring.· 1470nm will be used initially to provide a 1 Gbit/s point-to-point link between UCHG and NUIG. This

will provide a resilient connection when the MAN ring is complete.· 1490nm will be used to provide a 1 Gbit/s resilient link between GMIT and NUIG (upon ring closure)· 1510nm will be used to provide a 1 Gbit/s resilient link between GMIT and NUIG (upon ring closure)· 10G resilient link between NUIG and ESB Telecoms aggregating n * 1Gbit/s from Access Ring· 1530nm will be used to provide a 1 Gbit/s resilient link between NUIG and NUIG’s Research Institute

(upon ring closure)

1550nm

GMIT

1590nm

1490nm

3750 Switch

mux/demuxmux/demux

AdvaFSP

ESBT PoP

Cisco7606

3750 Stack

MAN Ring(n * 1Gbit/s)

10G

1Gig TX

Regional

National

CAO

3750 Switch

1510nm

NUIGResearch Institute

3750 Switch

1530nm

3750 Switch

1470nm

Transmission technologies

• We use a heterogeneous mix of technologies to achieve our goals:– CWDM in metropolitan areas gives us multiplexed fibre

services for short hops– DWDM between regions to give the same for longer hops– MPLS network to give multi-service offering– Multiple physical connections to add resiliency

25

HEAnet backbone network topology

How does it do it?• Circuits are managed using an automated

provisioning system• Based on client requirements, the system is

used to configure routers, service monitoring portal

• Aim to have client use the tool directly, with administrator later checking/approving circuit change requests

27

Network provisioning tool

The bigger picture

• NIBEST – NW link with NIRAN• UKERNA DF – SuperJANET 5 links• GÉANT – hybrid networking• Transit – preparing for the next step• INEX – peerings and more

29

INEX traffic growth

• HEAnet-INEX (IPv4)

• Total INEX

IP bandwidth

1

10

100

1,000

10,000

100,000

IP b

andw

idth

(Mbp

s)

Access b/w

External b/w

IP bandwidth

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

IP b

andw

idth

(Mbp

s)

Access b/w

External b/w

Summary of changesBefore After

•Network consists of managed links connecting regional POPs•Clients used managed links to connect to PoPs•Service monitored by HEAnet (IP) and telco providers (physical)

•More PoPs, connected by Dark Fibre•Clients used Dark Fibre and/or managed links to connect to PoPs•Service monitored by HEAnet (IP, Ethernet) and Fibre providers (physical)

Benefits

• Improved bandwidth• Guaranteed uncontended bandwidth service

levels• Reduced cost• Flexibility• Improved access to HEAnet services

35

Outline of servicesNETWORK

MANAGEMENTHOSTING

SERVICESMULTIMEDIA

APPLICATIONS

Network Operations Centre, Helpdesk

Co-location - equipment hosting

Hi-quality IP Video-conferencing

Up-to-minute traffic and performance

Web hosting, Database hosting, Email, Domains...

Video & audio streaming over

Internet

Spam Control

tools, MAPS

Anti-virus trends

Telco liaison, fault resolution and client

support

Hot Standby, failover website facility

High quality streamed content

(e.g. Oireachtas Live, IPTV)

Public Key Infrastructure

Authenticate Authorise

Accounting

Performance and quality maintenance

Mirroring, Multimedia content central

storage

Schools connectivity to backbone

SECURITY

Computer Emergency Response Team

Security Scanning Service

Phase 2 progress, 2nd Nov

Confirmed 7 Cancelled 4 In progress 17 Completed 13

Total number of links 43

37

Dark fibre statistics

• No. of DF providers 5• Aggregate length of DF pair 2,300 km• No. of primary client sites

with DF links 18 (of 40)• Total sites with DF connections 31

38

Next steps

• New model, service levels• Phase 3• Client service reviews• Review of research infrastructures• Integrated e-infrastructures

39

Questions?

Project history• In 2004, we commenced implementation of dark

fibre point to point links• Followed by additional of CWDM and DWDM

equipment to increase service possibilities• Commenced work on an automated provisioning

tool to create and manage configurations• Quickly grew into a heterogeneous network

combining a mix of technologies and providers

41

Scalable capacity• Backbone – 10 Gbit/s• Can add more 10 Gbit/s using existing

infrastructure• Client connections at 1 Gbit/s

42

What has been achieved

• High levels of service to clients- high bandwidth- resilience – path, equipment, service

• Scalability now and in the future• Underpins e-infrastructure• Complex and flexible• Rich mix of services• Secure

43

Trends since 1991

1

10

100

1,000

10,000

100,000

1,000,000

Jan-

91

Jan-

93

Jan-

95

Jan-

97

Jan-

99

Jan-

01

Jan-

03

Jan-

05

Valu

e re

lativ

e to

199

1

HEAnet traffic

Access bandwidth

External bandwidth

HEAnet staff