traffic matrices and its measurement
DESCRIPTION
Paolo Lucente, author of the software pmacct. Traffic matrices can greatly benefit key IP Service Provider activities like capacity planning, traffic engineering, better understand their traffic patterns and take meaningful peering decisions. This talk wants to present a way to build traffic matrices using telemetry data and BGP, leveraging along the way some case-studies with a technical cut. pmacct (http://www.pmacct.net) is a commonly used, free, open-source IPv4/IPv6a ccounting package which integrates a NetFlow/sFlow and a multi-RIB BGP collector in a single piece of software and is authored by the presenter.TRANSCRIPT
![Page 1: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/1.jpg)
Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
Traffic Matrices and its measurement
Paolo Lucente pmacct
![Page 2: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/2.jpg)
Agenda
o Introduction o The tool: pmacct o Setting the pitch o Case study: peering at AS286
Traffic Matrices and its measurement
Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
![Page 3: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/3.jpg)
Why speaking of traffic matrices?
– A;er focusing on traffic matrices during your course you can find this ques>on silly. But .. § In the industry, finding knowledgeable people on the subject is hard
§ Quite unspoken subject at technical conferences § Presenta>ons (nega>vely) stop to the math
![Page 4: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/4.jpg)
Why speaking of traffic matrices?
– So, a;er all, are traffic matrices useful to a network operator in the first place? Yes … § Capacity planning (build capacity where needed) § Traffic Engineering (steer traffic where capacity is available)
§ BeQer understand traffic paQerns (what to expect, without a crystal ball)
§ Support peering decisions (traffic insight, traffic engineering at the border, support what if scenarios)
![Page 5: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/5.jpg)
Review of traffic matrices: internal
– POP to POP, AR to AR, CR to CR
CR
CR
CR
CR
PoP
AR
AR
AR
AR
AR
PoP
AR
Customers Customers
AS1 AS2 AS3 AS4 AS5
Server Farm 1 Server Farm 2
© 2010 Cisco Systems, Inc./Cariden Technologies, Inc.
![Page 6: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/6.jpg)
Review of traffic matrices: external
– Router (AR or CR) to external AS or external AS to external AS (for IP transit providers)
CR
CR
CR
CR
PoP
AR
AR
AR
AR
AR
PoP
AR
Customers Customers
AS1 AS2 AS3 AS4 AS5
Server Farm 1 Server Farm 2
© 2010 Cisco Systems, Inc./Cariden Technologies, Inc.
![Page 7: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/7.jpg)
Let’s focus on an external traffic matrix to support peering decisions
– Analysis of exis>ng peers and interconnects: § Support policy and rou>ng changes § Fine-‐grained accoun>ng of traffic volumes and ra>os § Determine backbone costs associated to peering § Determine revenue leaks
– Planning of new peers and interconnects: § Who to peer next § Where to place next interconnect § Modeling and forecas>ng
![Page 8: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/8.jpg)
Our traffic matrix visualized
P P
P
PE A
PE D
PE C
PE B
P3
P1 P2
P4
BZ
A = { src_as, in_iface, peer_src_ip, peer_dst_ip, as_path, tstamp, bytes }
{ CZ, X (-‐> CY), PE C, PE A, AZ_AY, <>me>, <bytes> }
{ BX, Y (-‐> BY), PE B, PE C, CY_CX, <>me>, <bytes> }
{ AX, Z (-‐> AZ), PE A, PE B, BY_BZ, <>me>, <bytes> }
![Page 9: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/9.jpg)
Our traffic matrix shopping cart
– What is needed: § BGP § (IGP) § Telemetry data: NetFlow, sFlow § Collector infrastructure: tool, system(s) § Storage: either of RDBMS, RRD, noSQL § Maintenance and post-‐processing scripts
– Risks: § 800 pound gorilla project
![Page 10: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/10.jpg)
Agenda
o Introduction o The tool: pmacct o Setting the pitch o Case study: peering at AS286
Traffic Matrices and its measurement
Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
![Page 11: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/11.jpg)
pmacct is free open-‐source GPL’ed so;ware
pmacct
libpcap
BGP
maps
IS-IS
MySQL PostgreSQL SQLite 3
MongoDB BerkeleyDB
flat-‐files
memory tables
NetFlow IPFIX
sFlow
sFlow
NetFlow IPFIX
tee
![Page 12: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/12.jpg)
Why pmacct was started anyway?
– I was 23 – I had just joined CNR (Na>onal Council for Research) – I thought you can manage IP networks like that
![Page 13: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/13.jpg)
Why pmacct was started anyway?
– Before turning 24 I realized I was wrong: § Too much data (ie. flow-‐tools) and canned reports (ie. ntop) were causing lack of visibilty in my backbone.
§ Not everything was invented yet!
![Page 14: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/14.jpg)
Usage scenarios
![Page 15: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/15.jpg)
Key pmacct non-‐technical facts
§ A name you can’t spell a;er the second drink § 10 years old project § Free, open-‐source, independent § Under ac>ve development § Innova>on being introduced § Well deployed around, also large SPs § Aims to be the traffic accoun>ng tool closer to the SP community needs
![Page 16: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/16.jpg)
Some technical facts
§ Pluggable architecture • Straighnorward to add support for new collec>on methods or backends
§ An abstrac>on layer allows out-‐of-‐the-‐box any collec>on method to interact with any backend
§ Both mul>-‐process and (coarse) mul>-‐threading • Mul>ple plugins (of same or different type) can be instan>ated at run>me, each with own config
![Page 17: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/17.jpg)
Introducing BGP na>vely into a NetFlow/sFlow collector
– pmacct introduced a Quagga-‐based BGP daemon § Implemented as a parallel thread within the collector § Doesn’t send UPDATEs and WITHDRAWs whatsoever § Behaves as a passive BGP neighbor § Maintains per-‐peer BGP RIBs § Supports 32-‐bit ASNs; IPv4 and IPv6 families
– Why BGP at the collector? § Telemetry reports on forwarding-‐plane § Telemetry should not move control-‐plane informa>on over and over
![Page 18: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/18.jpg)
Agenda
o Introduction o The tool: pmacct o Setting the pitch o Case study: peering at AS286
Traffic Matrices and its measurement
Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
![Page 19: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/19.jpg)
Getng BGP to the collector – Let the collector BGP peer with all PE devices: facing peers, transit and customers.
– Determine memory footprint (below in MB/peer)
44.03
22.73
19.97 18.59 18.12 17.89 17.76 17.57 17.48 17.39
50
0
10
20
30
40
50
60
0 200 400 600 800 1000 1200 1400
MB/peer >= 0.12.4
MB/peer < 0.12.4
Number of BGP peers
MB/pe
er
500K IPv4 routes, 50K IPv6 routes, 64-‐bit executable
~ 9GB total memory @ 500 peers
![Page 20: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/20.jpg)
– Set the collector as iBGP peer at the PE devices: § Configure it as a RR client for best results § Collector acts as iBGP peer across (sub-‐)ASes
– BGP next-‐hop has to represent the remote edge of the network model: § Typical scenario for MPLS networks § Can be followed up to cover specific scenarios like:
• BGP confedera>ons – Op>onally polish the AS-‐Path up from sub-‐ASNs
• default gateway defined due to par>al or default-‐only rou>ng tables
Getng BGP to the collector (cont.d)
![Page 21: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/21.jpg)
Getng telemetry to the collector
– Export ingress-‐only measurements at all PE devices: facing peers, transit and customers. § Traffic is routed to des>na>on, so plenty of informa>on on where it’s going to
§ It’s crucial instead to get as much as possible about where traffic is coming from
– Leverage data reduc>on techniques at the PE: § Sampling § Aggrega>on (but be sure to carry IP prefixes!)
![Page 22: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/22.jpg)
Telemetry data/BGP correla>on
![Page 23: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/23.jpg)
Storing data persistently – Data need to be aggregated both in spa>al and temporal dimensions before being wriQen down: § Op>mal usage of system resources § Avoids expensive consolida>on of micro-‐flows § Suitable for project-‐driven data-‐sets
– Open-‐source RDBMS appear a natural choice § Able to handle large data-‐sets § Flexible and standardized query language § Solid and scalable (clustering, par>>oning)
– noSQL DBs picking up: between myth and bust ..
![Page 24: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/24.jpg)
Briefly on scalability – A single collector might not fit it all:
§ Memory: can’t store all BGP full rou>ng tables § CPU: can’t cope with the pace of telemetry export § Divide-‐et-‐impera approach is valid:
• Assign PEs (both telemetry and BGP) to collectors • Assign collectors to RDBMSs; or cluster the RDBMS.
– The matrix can get big, but can be reduced: § Keep smaller routers out of the equa>on § Filter out specific services/customers on dense routers § Focus on relevant traffic direc>on (ie. upstream if CDN, downstream if ISP)
§ Sample or put thresholds on traffic relevance
![Page 25: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/25.jpg)
Agenda
o Introduction o The tool: pmacct o Setting the pitch o Case study: peering at AS286
Traffic Matrices and its measurement
Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
![Page 26: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/26.jpg)
Case-‐study: peering at AS286
Nego>a>on
Engineering
Produc>on
Op>miza>on
– Peering as a cycle – NetFlow + BGP traffic matrix
steers peering op>miza>on: § Iden>fy new and “old” peers § Traffic analysis: backbone
costs, 95th percen>les, ra>os § Analysis of interconnec>on
density and traffic dispersion § Forecas>ng and trending § Ad-‐hoc queries from Design
& Engineering and indeed … the IPT Product Manager
![Page 27: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/27.jpg)
Case-‐study: peering at AS286 – 250+ Gbps rou>ng-‐domain – 100+ high-‐end routers around
the globe: § Export sampled NetFlow § Adver>se full rou>ng table § Mix of Juniper and Cisco
– Collector environment: § Runs on 2 Solaris/SPARC zones § pmacct: dual-‐core, 4GB RAM § MySQL: quad-‐core, 24GB RAM,
500 GB disk – Data reten>on: 6 months
AS286 rou>ng domain
pmacct
MySQL
Internal users
![Page 28: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/28.jpg)
– AS286 backbone routers are first configured from templates: § NetFlow + BGP collector IP address defined over there § Enabler for auto-‐discovery of new devices
– Edge interfaces are provisioned following service delivery manuals: § Relevant manuals and TSDs include NetFlow ac>va>on § Periodic checks NetFlow is ac>ve where it should
– Maps, ie. source peer-‐AS, are re-‐built periodically
Case-‐study: peering at AS286
![Page 29: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/29.jpg)
Thanks for your attention! Questions? Now or later ..
Paolo Lucente pmacct
<paolo at pmacct dot net> Keep in touch via LinkedIn
Traffic Matrices and its measurement
Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
![Page 30: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/30.jpg)
Backup slides
Traffic Matrices and its measurement
Universitat Politecnica de Catalunya, Castelldefels – Jun, 5th 2013
![Page 31: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/31.jpg)
IGP (IS-‐IS) integra>on
– A Quagga-‐based IS-‐IS daemon was introduced: § Implemented as a parallel thread within the collector § IS-‐IS neighborship over a GRE tunnel § Currently limited to single neighborhip, single level, single topology
§ Useful to look up non BGP-‐routed networks § Promising, many plans:
• Implement a non real-‐>me, IGP-‐protocol agnos>c version • LFA kind of computa>ons • Collec>ng feedback on interest about TRILL
![Page 32: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/32.jpg)
Storing data persisently: RDBMS create table acct_bgp ( agent_id INT(4) UNSIGNED NOT NULL,
as_src INT(4) UNSIGNED NOT NULL,
as_dst INT(4) UNSIGNED NOT NULL, peer_as_src INT(4) UNSIGNED NOT NULL, peer_as_dst INT(4) UNSIGNED NOT NULL, peer_ip_src CHAR(15) NOT NULL, peer_ip_dst CHAR(15) NOT NULL, comms CHAR(24) NOT NULL,
as_path CHAR(21) NOT NULL, local_pref INT(4) UNSIGNED NOT NULL, med INT(4) UNSIGNED NOT NULL, packets INT UNSIGNED NOT NULL,
bytes BIGINT UNSIGNED NOT NULL,
stamp_inserted DATETIME NOT NULL,
stamp_updated DATETIME,
PRIMARY KEY (…)
);
BGP Fields
Counters
Time
Tag
shell> cat peers.map id=65534 ip=X in=A id=65533 ip=Y in=B src_mac=J id=65532 ip=Z in=C bgp_nexthop=W [ … ]
shell> cat pretag.map id=100 peer_src_as=<customer> id=80 peer_src_as=<peer> id=50 peer_src_as=<IP transit> [ … ]
– In any schema (a subset of) BGP primi>ves can be freely mixed with (a subset of) L1-‐L7 primi>ves
![Page 33: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/33.jpg)
Storing data persisently: RDBMS – Traffic delivered to a BGP peer, per loca>on: mysql> SELECT peer_as_dst, peer_ip_dst, SUM(bytes), stamp_inserted
FROM acct_bgp
WHERE peer_as_dst = <peer | customer | IP transit> AND
stamp_inserted = < today | last hour | last 5 mins >
GROUP BY peer_as_dst, peer_ip_dst; – Aggregate AS PATHs to the second hop: mysql> SELECT SUBSTRING_INDEX(as_path, ‘.’, 2) AS as_path, bytes
FROM acct_bgp
WHERE local_pref = < IP transit pref> AND
stamp_inserted = < today | yesterday | last week >
GROUP BY SUBSTRING_INDEX(as_path, ‘.’, 2)
ORDER BY SUM(bytes);
– Focus peak hour (say, 8pm) data: mysql> SELECT … FROM … WHERE stamp_inserted LIKE ‘2010-02-% 20:00:00’
…
![Page 34: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/34.jpg)
Storing data persisently: RDBMS – Traffic breakdown, ie. top N grouping BGP peers of the same kind (ie. peers, customers, transit):
mysql> SELECT … FROM … WHERE …
local_pref = <<peer | customer | IP transit> pref>
…
– Download traffic matrix (or a subset of it) to 3rd party backbone planning/traffic engineering applica>on (ie. Cariden, Wandl, etc.):
mysql> SELECT peer_ip_src, peer_ip_dst, bytes, stamp_inserted
FROM acct_bgp
WHERE [ peer_ip_src = <location A> AND
peer_ip_dst = <location Z> AND … ]
stamp_inserted = < today | last hour | last 5 mins >
GROUP BY peer_ip_src, peer_ip_dst;
![Page 35: Traffic Matrices and its measurement](https://reader030.vdocument.in/reader030/viewer/2022020217/54b4ebb24a795994458b4652/html5/thumbnails/35.jpg)
Storing data persisently: MongoDB
– pmacct opening to noSQL databases – noSQL landscape difficult to move through, ie. fragmented and lacks of standardiza>on
– MongoDB seemed interes>ng for: § Its na>ve grouping opera>on (more performing and less complex than map/reduce)
§ Its horizontal scaling concept (called sharding) – Currently collec>ng feedbck: some preliminar tes>ng at one large operator looks promising ..