traffic analysis with netflow - sinog
TRANSCRIPT
![Page 1: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/1.jpg)
SINOG 3.0 meeting, Ljubljana – Jun 2016
pmacct and streaming telemetry
Paolo Lucente
pmacct
![Page 2: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/2.jpg)
whoami: Paolo
Been originally working for operators for a while
Been working for vendors for a little while after that
Been involved with IP accounting for a while• Hence I stumbled upon NetFlow in the 90’s
Within operators, network traffic telemetry is beneficial in several contexts, ie.:• Traffic engineering
• Capacity planning
• Peering
• …
• and also (ie. not only) security
![Page 3: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/3.jpg)
libpcap
pmacct is open-source, free, GPL’ed software
sFlow
BGP
maps
IGP
MySQL
PgSQL
SQLite
MongoDB
BerkeleyDB
flat-files
RabbitMQ
Kafka
memory
tables
sFlow
tee
NetFlow
IPFIX
NetFlow
IPFIX
http://www.pmacct.net/
streaming
telemetry
BMP
![Page 4: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/4.jpg)
pmacct: a few simple use-cases
BMP
flat-files
tee
NetFlow
IPFIX
sFlow
Kafka
IPFIXlibpcap
![Page 5: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/5.jpg)
pmacct: one slightly more complex use-case
BGP
flat-files
tee
NetFlow
IPFIX
Kafka
MySQL
aggregation method #1
aggregation method #2
nfacctd
![Page 6: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/6.jpg)
Usage scenarios
![Page 7: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/7.jpg)
Key pmacct non-technical facts
10+ years old project
Can’t spell the name after the second drink
Free, open-source, independent
Under active development
Innovation being introduced
Well deployed around, also large SPs
Aims to be the traffic accounting tool closer to the SP community needs
![Page 8: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/8.jpg)
Some technical facts (1/2)
Pluggable architecture:• Can easily add support for new data sources and backends
Correlation of data sources:• Natively supported data sources (ie. BGP, BMP, IGP,
streaming telemetry)
• External data sources via tags and labels
Pervasive data-reduction techniques, ie.:• Data aggregation
• Filtering
• Sampling
![Page 9: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/9.jpg)
Some technical facts (2/2)
Build multiple views out of the very same collected network traffic dataset , ie.:• Unaggregated to flat-files for security and forensics; or to
message brokers (RabbitMQ, Kafka) for Big Data
• Aggregated as [ <ingress router>, <ingress interface>, <BGP next-hop>, <peer destination ASN> ] and sent to a SQL DB to build an internal traffic matrix for capacity planning purposes
Enable analytics against the collected data sources (ie. BGP, BMP, streaming telemetry):• Stream real-time
• Dump at regular time intervals (possible state compression)
![Page 10: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/10.jpg)
Summarizing Cisco IOS-XR Telemetry Configuration Guide (at the time of this writing):• Streaming telemetry lets users direct data to a
configured receiver
• This is achieved by leveraging the capabilities of M2M communication
• The data is used by DevOps people to optimize networks by collecting analytics of the network in real-time
Streaming telemetry
![Page 11: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/11.jpg)
flat-files
streaming
telemetryKafka
Telemetry dump at regular time intervals
pmtelemetryd
Telemetry real-time log
pmacct & streaming telemetry (1/2)
![Page 12: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/12.jpg)
Streaming
telemetry
flat-files
tee
NetFlow
IPFIX
Kafka
MySQL
aggregation method #1
aggregation method #2
nfacctd
pmacct & streaming telemetry (2/2)
![Page 13: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/13.jpg)
<rant>
![Page 14: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/14.jpg)
Streaming telemetry
Been so far an exciting experience of delving into an enchanted, non standardized world:
• Data modelling is cool:
Standardization focuses on this part
• Transport, subscription mechanisms, data serialization are not cool enough:
Data is known to spontaneously migrate
And then get magically decoded
Things like that, “details” ..
![Page 15: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/15.jpg)
Streaming telemetry
Having myself deep roots in the Service Providers community, I do believe in the mantra “Operators should get more involved in standardization”
But now look at:
• http://www.openconfig.net/projects/streaming-telemetry/
• http://www.openconfig.net/about/participants/
• This does feel a bit like revenge, doesn’t it?
![Page 16: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/16.jpg)
Streaming telemetry
Homework: figure out your own practical examples when it comes to “details” (some keywords as hint: gRPC, netconf, restconf, JSON, GPB, Avro)
This is all with still little adoption (maybe PoC’s?) outside the circle of the Big Guys
“Let’s hope they don’t turn out into the enterprise MIBs of the 21st century” (cit. David Barroso)
![Page 17: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/17.jpg)
How is
![Page 18: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/18.jpg)
A peaceful gathering of Vendors
![Page 19: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/19.jpg)
![Page 20: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/20.jpg)
(as in any worse)
![Page 21: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/21.jpg)
than
![Page 22: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/22.jpg)
An Operators (only!) Working Group
![Page 23: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/23.jpg)
?
![Page 24: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/24.jpg)
( Btw, this is a rare picture of Vendors holding breath during an Operators Working Group meeting )
![Page 25: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/25.jpg)
Streaming telemetry
Steaming telemetry has great potential
For some aspects of it, fragmentation flag is on
Fragmentation as in: “several equivalent choices”
Who benefits from fragmentation?
Let’s not take abstraction as the excuse
![Page 26: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/26.jpg)
</rant>
![Page 27: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/27.jpg)
Further information about pmacct
https://github.com/pmacct/pmacct
• Official GitHub repository, where star and watch us
http://www.pmacct.net/lucente_pmacct_uknof14.pdf
• More about coupling telemetry and BGP
http://ripe61.ripe.net/presentations/156-ripe61-bcp-planning-and-te.pdf
• More about traffic matrices, capacity planning & TE
http://wiki.pmacct.net/ImplementationNotes
• Implementation notes (RDBMS, maintenance, etc.)
![Page 28: Traffic analysis with NetFlow - SINOG](https://reader035.vdocument.in/reader035/viewer/2022080222/62e87f5515d59b326f064a1b/html5/thumbnails/28.jpg)
Thanks! Questions?
Paolo Lucente <[email protected]>
SINOG 3.0 meeting, Ljubljana – Jun 2016
pmacct and streaming telemetry