ixp network monitoring tool portability: possible project? · ixp network monitoring tool...

IXP network monitoringtool portability:possible project?

John SouterLondon Internet Exchange

Presentation to UCL seminar

IXP networkmonitoring

toolportability:possibleproject?

25/11/03

John Souter

Agenda

� About LINX� About Euro-IX� Network monitoring

� Where we are today� Where I want us to be

� Project idea� Questions & discussion

Jargon note: IXP = Internet Exchange Point



25/11/03

John Souter

But first a quiz…..

� Who has heard of LINX?� Who has heard of Internet

Exchange Points?� Where you curious about how

ISPs connect together?



25/11/03

John Souter

LINX History

� Established in1994 by 5 memberISPs� Co-operation amongst competitors

� Objective: to keep UK traffic in UK� Housed in external independent

facilities� Telehouse North (TFM6)

� Very simple back then!



25/11/03

John Souter

LINX Now

� Steady linear growth to ~140 members� Throughput up to ~96% of UK traffic� Still a mutual organisation

� “Not open to capture”� Completely open, the only rules are

technical ones� Points of Presence in 8 co-lo sites

throughout Docklands & City� Extensive & diverse multi-vendor dark

fibre network� Offices in Peterborough and London



25/11/03

John Souter

LINX traffic

https://stats.linx.net/

10.45amtoday

https://stats.linx.net/



25/11/03

John Souter

LINX architecture



25/11/03

John Souter

Benefits of LINX membership

� Tangible benefits:� Peering saves on transit costs� Dual switch vendor architecture on

separate LAN’s = Very high resilience� Performance: downtime negligible

� Intangible benefits:� Pioneering new technology

• First IXP ever to run 1GigE and then 10GigEover a metropolitan network

� Voice of the industry in UK & increasinglyEU regulatory affairs

� Provides a forum for direct (social)networking



25/11/03

John Souter

Peering?

� Peering is the arrangement oftraffic exchange between ISPs

� Peering is usually settlement free� It is a voluntary arrangement

� LINX does not force peering

� It works because both memberssave on transit costs

� How?



25/11/03

John Souter

Before and After

� Both parties pay transit

� Upstream wins £££

� Still some residual transit

� How much can be offloadedat LINX ?



25/11/03

John Souter

Break Even

LINX Break Even

0100002000030000400005000060000700008000090000

100000

5 15 25 35 45 55 65 75

Traffic Mb

Cos

t £ LINX COSTSTRANSIT COSTS

Break Even

0100002000030000400005000060000700008000090000

100000

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Traffic Mb

Cost

£ LINXTRANSIT

� It’s different for everybody….� Graphs assume £100 per meg per month� 75% and 50% offload respectively� What’s routes are available at LINX ?



25/11/03

John Souter

Some Figures

UK57%

US9%

RO1%DE

5%

NL7%IE

1%FR3%

SE1%

CH2%

SG1%

JP1%

ES3%

IT1%

KR1%

HK1%

NO1%

BE3%

DK1% FI

2%

UK US RO DE NL IE FR SE CH SG JP ES IT KR HK FI NO BE DK

� 57% UK | 9% US | 5% DE | 7% NL 3% ES, FR & BE

� 8% Rest of Europe, 5% Rest of World

� Approx 50% Global routing table available



25/11/03

John Souter

LINX distinctive features

(note: versus other European IXPs)� Dual LAN architecture� NCAP (regulatory) activity� Private interconnect within the

exchange� Traffic-based billing� Also:

� Multiple sites, staffing, ambition,plans….



25/11/03

John Souter

LINX approach to resilience

� Dual LAN architecture with dual switchvendors

� Own dark fibre with multiple diverserouting between sites

� Hygiene rules (in the MoU) with strictmonitoring

� Conservative approach to engineeringconfiguration, backed up by roadmaplook-ahead

� Multiple co-location sitesSummary:

� heavy over-engineering to prevent failure



25/11/03

John Souter

Significance of LINX to the UK

� LINX has become part of the UK’scritical national infrastructure

� LINX members achieve efficientpeering, so end users benefit by:� Efficient connectivity� High resilience� Low latency

� LINX is not subject to the whimsof the stock market!



25/11/03

John Souter

European IXPs

http://www.dix.dk/euro/

http://www.dix.dk/euro/



25/11/03

John Souter

Models for IXPs (in Europe)

� Mutual model (member owned)• E.g. LINX, AMS-IX, MIX, MaNAP, LoNAP

� ‘Service of’� Service of a commercial organisation

• E.g. ESPANIX, DE-CIX, PARIX

� Service of an academic/research network• E.g. VIX, BNIX, CATNIX

� Fully commercial, for profit• E.g. XchangePoint, LIPEX

� USA IXP scene is quite different…..



25/11/03

John Souter

Euro-IX

� Founded by 7 exchanges (incLINX) as a European IXP tradeassociation

� Now has 27 members� Active projects on various topics� Helping the ‘industry’ grow up and

learn from each other� See http://www.euro-ix.net

http://www.euro-ix.net/



25/11/03

John Souter

Euro-IX today

� The association is still growing(27 members from 19 countries)

� Total combined traffic >73.6Gb/s� Source: public stats from 21 IXPs

� Portal lists 1,270 ASNs in the Euro-IXmember/customer database� Up from 807 in Sept 2002

� Value of the portal is growing:� Page views up from 4,700 to 6,000/month� 1,400 unique IPs./month



25/11/03

John Souter

Network monitoring at LINX

� 3 years ago:� Very simple system� Built internally with NOCOL

(open source tool)� Then, on 16 October 2001:

� Big, big outage!� Unmeshed LANs…..� Monitoring system inadequate

� Project to redevelop the system



25/11/03

John Souter

New LINX monitoring system

� New system is made up of:� syslog-ng� SNIPS polling/monitoring system� RRD� SMStools SMS messaging system� Alarm Processor� ‘CentraLINX’ web interface

� Dedicated monitoring servers incore LINX sites



25/11/03

John Souter

Distributed Monitoring Servers

� Ping monitor members directly fromthe Peering LAN

� Monitoring across the exchangebetween sites

� Allows us to see traffic from differentsites. Visibility of both Extreme andFoundry networks and any VLAN

� Collect syslog/trap data locally to eachsite and relay to central log host

� Possibility to assist members withtraffic problems by mirroring ports.



25/11/03

John Souter

New Alarm Processor

� It glues syslog, SNIPS and othersystems together

� The “brain” behind the system� Enriches the alarm information� Correlates and classifies events

and decides what to do with them� Some dependency checks� Suppress downstream alarms



25/11/03

John Souter

New Alarm Processor

� Suppress if: -� the operator told us to shut up!� known error (ticket open)� the NMS itself is affected� if dependency check fails� duplicate alarm� state flapping� multiple events for the same port� a more serious condition exists� …..



25/11/03

John Souter

Internal Network

VLAN3Each LINX site had a site router

Which connected to the restwith a VLAN



25/11/03

John Souter

Internal Network

Foundry VLAN3 Extreme VLAN3

When the network split, weduplicated the connections to

both sides



25/11/03

John Souter

Foundry VLAN3 Extreme VLAN3

192.168.x.x

Sniffer Servermoni tor ing /analysis Monitoring Agent

Internal Network



25/11/03

John Souter

192.168.x.x

ExtremeFoundry

VLAN1VLAN3

VLAN1VLAN3

Ping

Alarms

Sniffer Servermonito ring /analy sis

Monitoring Agent

syslog195.66.224.0/23 195.66.226.0/23

Member

Member

Member

Member

Member

Member 195.66.232.x

Internal Network



25/11/03

John Souter

192.168.x.x

ExtremeFoundry

VLAN1VLAN3

VLAN1VLAN3


Monitoring Agent

195.66.224.0/23 195.66.226.0/23

Member

Member

Member

Member

Member

Member 195.66.232.x

Syslo

g:po

rtX

down

!Alarms to

central server Can’t p

ingIP

195.6

6.226

.x

BGP session with collector down

Member down event

192.168.x.x

ExtremeFoundry

VLAN1VLAN3

VLAN1VLAN3


Monitoring Agent

195.66.224.0/23 195.66.226.0/23

Member

Member

Member

Member

Member

Member 195.66.232.x

Sysl

og: p

ort X

dow

n!

Alarms to


ingIP

195.6

6.226

.x

BGP session with collector down



25/11/03

John Souter

192.168.x.x

ExtremeFoundry

VLAN1VLAN3

VLAN1VLAN3


Monitoring Agent

195.66.224.0/23 195.66.226.0/23

Member

Member

Member

Member

Member

Member 195.66.232.xBGP session with collector down

Alar

ms

toce

ntra

l ser

ver

syslog: ISL to other switch down!!!!!!!Can’t ping 195.66.224.xxx

Switch down events



25/11/03

John Souter

192.168.x.x

ExtremeFoundry

VLAN1VLAN3

VLAN1VLAN3


Monitoring Agent

195.66.224.0/23 195.66.226.0/23

Member

Member

Member

Member

Member

Member 195.66.232.xGah! A module went away.I’m not very happy now.Pain in all the diodes down my left side...

IXP-WATCH: Hmm. Something oddis going on here. Too many arps….

syslog….

??

Alar

ms

toce

ntra

l ser

ver

??

If the switch is loggingerrors, member alarms

suppressed for 5 minutes.

Alarm processor filters outspurious or “normal” errorsand alerts about bad things.

Truckloads of messagessuppressed

Switch errors



25/11/03

John Souter

192.168.x.x

ExtremeFoundry

VLAN1VLAN3

VLAN1VLAN3

Ping

Alarms


Monitoring Agent

syslog195.66.224.0/23 195.66.226.0/23

Member

Member

Member

Member

Member

Member 195.66.232.x

Sysl

og: p

ort X

dow

n!

Alarms to


ingIP

195.6

6.226

.x

BGP session with collector downBGP session with collector down

Gah! A module went away.I’m not very happy now.Pain in all the diodes down my left side...

Alar

ms

toce

ntra

l ser

ver

syslog: ISL to switch down!

!!!!!!Can’t ping 195.66.224.xxx

Can’t ping switchxx

IXP-WATCH: Hmm. Something oddis going on here. Too many arps….

syslog….

??

Alar

ms

toce

ntra

l ser

ver

??

Alarm meltdown….



25/11/03

John Souter

But…..

� Alarm processor is an in-housedevelopment (Perl script)

� Data formats are unique to LINX� Sharing our tool with others is

difficult because of this� We are still monitoring ‘outages’

� Would like to monitor trends andidentify potential outages beforethey happen

� More development work to do…..



25/11/03

John Souter

Euro-IX situation on monitoring

� Commonality amongst the devicesused by IXPs:� switches from Cisco, Foundry & Extreme� routers mainly from Cisco (and perhaps

Juniper)� mostly UNIX/Linux servers

� Most European IXPs have similarset-ups, with notions of:� 'members‘� 'connection ports‘� allocated IP addresses etc.

� Some are developing tools to share



25/11/03

John Souter

Project idea

� Define an abstract/intermediateformat or API for IXP networkconfiguration� whichever seems the better approach

� Explore with Euro-IX the adoptionof this as a 'standard'



25/11/03

John Souter

Project deliverables?

� A standard format or APIdefinition

� Trial implementations for LINXand say one other co-operativeIXP

� Recommendations for usage



25/11/03

John Souter

Co-operation possibilities

� Euro-IX tools forum keen on theidea

� Several IXPs have said theywould co-operate:� AMS-IX� MIX� Possibly others…..

� This would be mostly by email,given geographical issues



25/11/03

John Souter

Questions & discussion

ixp network monitoring tool portability: possible project? · ixp network monitoring tool...

Documents