pmacct - nlnog...aggregation method #1 aggregation method #2 nfacctd usage scenarios key pmacct...
TRANSCRIPT
![Page 1: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/1.jpg)
NLNOG Day 2016, Amsterdam – Sep 2016
pmacct and Streaming Telemetry
Paolo Lucente
pmacct
![Page 2: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/2.jpg)
whoami: Paolo
Been originally working for operators for a while
Been working for vendors for a little while after that
Been involved with IP accounting for a while
• Hence I stumbled upon NetFlow i the 90’s
Within operators, network traffic telemetry is beneficial in several contexts, ie.:
• Traffic engineering
• Capacity planning
• Peering
• …
• and also (ie. not only) security
![Page 3: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/3.jpg)
libpcap
pmacct is open-source, free, GPL’ed software
sFlow
BGP
maps
IGP
MySQL
PgSQL
SQLite
MongoDB
BerkeleyDB
flat-files
RabbitMQ
Kafka
memory
tables
sFlow
tee
NetFlow
IPFIX
NetFlow
IPFIX
http://www.pmacct.net/
Streaming
Telemetry
BMP
![Page 4: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/4.jpg)
pmacct: a few simple use-cases
BMP
flat-files
tee
NetFlow
IPFIX
sFlow
Kafka
IPFIX libpcap
![Page 5: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/5.jpg)
pmacct: one slightly more complex use-case
BGP
flat-files
tee
NetFlow
IPFIX
Kafka
MySQL
aggregation method #1
aggregation method #2
nfacctd
![Page 6: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/6.jpg)
Usage scenarios
![Page 7: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/7.jpg)
Key pmacct non-technical facts
10+ years old project
Ca ’t spell the a e after the se o d dri k
Free, open-source, independent
Under active development
Innovation being introduced
Well deployed around, also large SPs
Aims to be the traffic accounting tool closer to
the SP community needs
![Page 8: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/8.jpg)
Some technical facts (1/2)
Pluggable architecture:
• Can easily add support for new data sources and backends
Correlation of data sources:
• Natively supported data sources (ie. BGP, BMP, IGP,
Streaming Telemetry)
• External data sources via tags and labels
Pervasive data-reduction techniques, ie.:
• Data aggregation
• Filtering
• Sampling
![Page 9: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/9.jpg)
Some technical facts (2/2)
Build multiple views out of the very same collected network traffic dataset , ie.: • Unaggregated to flat-files for security and forensics; or to
message brokers (RabbitMQ, Kafka) for Big Data
• Aggregated as [ <ingress router>, <ingress interface>, <BGP next-hop>, <peer destination ASN> ] and sent to a SQL DB to build an internal traffic matrix for capacity planning purposes
Enable analytics against the collected data sources (ie. BGP, BMP, Streaming Telemetry): • Stream real-time
• Dump at regular time intervals (possible state compression)
![Page 10: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/10.jpg)
Summarizing Cisco IOS-XR Telemetry Configuration Guide (at the time of this writing):
• Streaming Telemetry lets users direct data to a configured receiver
• This is achieved by leveraging the capabilities of M2M communication
• The data is used by DevOps people to optimize networks by collecting analytics of the network in real-time
Streaming Telemetry
![Page 11: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/11.jpg)
flat-files
Streaming
Telemetry Kafka
Telemetry dump at regular time intervals
pmtelemetryd
Telemetry real-time log
pmacct & Streaming Telemetry (1/2)
![Page 12: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/12.jpg)
Streaming
telemetry
flat-files
tee
NetFlow
IPFIX
Kafka
MySQL
aggregation method #1
aggregation method #2
nfacctd
pmacct & Streaming Telemetry (2/2)
![Page 13: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/13.jpg)
<rant>
![Page 14: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/14.jpg)
Streaming Telemetry
Been so far an exciting experience of delving
into an enchanted, non standardized world:
• Data modelling is cool:
Standardization focuses on this part
• Transport, subscription mechanisms, data
serialization are not cool enough apparently:
Data is known to spontaneously migrate
And then get magically decoded
Thi gs like that, details ..
![Page 15: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/15.jpg)
Streaming Telemetry
Having myself deep roots in the Service Providers
o u it , I do elie e i the a tra Operators
should get more involved in standardization
But now look at:
• http://www.openconfig.net/projects/streaming-telemetry/
• http://www.openconfig.net/about/participants/
• This does feel a it like re e ge, does ’t it?
![Page 16: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/16.jpg)
Streaming Telemetry
Homework: figure out your own practical e a ples he it o es to details (so e keywords as hint: gRPC, netconf, restconf, JSON, GPB, Avro)
Fun fact: GPB requires inclusion of source code to work: was it not that when you do that, licensing of code starts to kick in?
Quote fro the i dustr : Let’s hope the do ’t turn out into the enterprise MIBs of the 21st e tur (Da id Barroso)
![Page 17: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/17.jpg)
Streaming Telemetry
This is all with still little adoption (maybe PoC’s?)
outside the circle of the Big Guys
![Page 18: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/18.jpg)
How is
![Page 19: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/19.jpg)
A peaceful gathering of Vendors
![Page 20: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/20.jpg)
![Page 21: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/21.jpg)
(as in any worse)
![Page 22: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/22.jpg)
than
![Page 23: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/23.jpg)
An Operators (only!) Working Group
![Page 24: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/24.jpg)
?
![Page 25: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/25.jpg)
( Btw, this is a rare picture of Vendors holding breath
during an Operators Working Group meeting )
![Page 26: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/26.jpg)
Streaming Telemetry
Steaming Telemetry has great potential
For some aspects of it, fragmentation flag is on
Frag e tatio as i : se eral equivalent hoi es
Who benefits from fragmentation?
Let’s ot take a stra tio as the e use
![Page 27: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/27.jpg)
</rant>
![Page 28: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/28.jpg)
Further information about pmacct
https://github.com/pmacct/pmacct
• Official GitHub repository, where star and watch us
http://www.pmacct.net/lucente_pmacct_uknof14.pdf
• More about coupling telemetry and BGP
http://ripe61.ripe.net/presentations/156-ripe61-bcp-
planning-and-te.pdf
• More about traffic matrices, capacity planning & TE
http://wiki.pmacct.net/ImplementationNotes
• Implementation notes (RDBMS, maintenance, etc.)
![Page 29: pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent](https://reader033.vdocument.in/reader033/viewer/2022060906/60a15b5548dd821d3c6e57b8/html5/thumbnails/29.jpg)
Thanks! Questions?
Paolo Lucente <[email protected]>
pmacct and Streaming Telemetry
NLNOG Day 2016, Amsterdam – Sep 2016