![Page 1: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/1.jpg)
Large scale passive monitoring at 10Gbps on commodity hardware
Campus network monitoring and security workshop April 24, 2014 Arne Øslebø, [email protected]
![Page 2: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/2.jpg)
UNINETT monitoring infrastructure
4/25/14 SLIDE 2
![Page 3: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/3.jpg)
Original hardware setup
Opticalsplitter
Passive monitoring card: Endace or Napatech GPS
UNINETT
![Page 4: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/4.jpg)
Active and passive monitoring
Appflow
Mping
Multicast Beacon
Throughput test
![Page 5: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/5.jpg)
Original Appflow architecture
SQL database Frontend
Collectors
Probes
IPFIX flow recordsto anycast address
YAFPackets Flow records
![Page 6: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/6.jpg)
10Gbps challengesTheoretical packet rate: 14.88 million pps
0
2000
4000
6000
8000
10000
12000
14000
16000
Flows per second
# o
f flo
ws
Number of flows
![Page 7: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/7.jpg)
New Appflow architecture
SQL database Frontend
Collectors
Probes
IPFIX flow recordsto anycast address
IPFIX aggregation recordsto anycast address
IPFIXexporter
Packets
Aggregator
Flow records
Aggregatedrecords
![Page 8: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/8.jpg)
TILEempower
● Based on TILERA cpu– Up to 72 cores
● Pros– Good performance– Special instructions for
packet processing– Very good documentation– DPI library
● Cons– Difficult to program– Price
![Page 9: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/9.jpg)
Intel X520 family of NICs
● Designed for virtualization● Support multi-core processors
– Hardware based load balancing● DMA transfer of captured packets● Hardware counters● Supports both 1 and 10 Gbps
![Page 10: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/10.jpg)
Drivers for Intel X520
● Standard drivers not very good for passive monitoring– Too many interrupts per second
● Packet I/O Engine– No longer maintained– http://shader.kaist.edu/packetshader/io_engine/
● netmap - a novel framework for fast packet I/O– Originally developed for FreeBSD– Unstable port to Linux– http://info.iet.unipi.it/~luigi/netmap/
● PF_RING with DNA– Stable and well maintained– Multiple applications can access same buffer– Not GPL, but free for academic use– http://www.ntop.org/products/pf_ring/dna/
![Page 11: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/11.jpg)
Server hardware
● Dell PowerEdge R620● CPU: Intel Xeon E5-2690, 2.9GHz, 8 cores , hyper-threading
– Support for second CPU● 32GB 1600MHz RDIMM● Intel X520DP
– Two ports with pluggable SFP+
![Page 12: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/12.jpg)
Packet capture performance
Gbps Mpps Cpu load (%)
Packet drop(%)
0.7 1 1 0
3.3 5 4 0
6.7 10 7 0
10.2 15 13 0
13.9 20 18 0
16.8 25 23 0
20 29.8 31 3.2
64 bytes packet size, two ports, one core
Gbps Mpps CpuLoad(%)
Packet drop (%)
17.3 5 7 0
20 6.5 9 0
Realistic packet size distribution, two ports, 8 cores for each port
![Page 13: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/13.jpg)
nProbe
● An Extensible NetFlow v5/v9/IPFIX GPL Probe for IPv4/v6● http://www.ntop.org/products/nprobe/● Good performance● Well maintained● Large user base● Multi-threaded
– But recommends running multiple single-thread instances● IP tagging
– AS numbers, countries– MaxMind: http://dev.maxmind.com/geoip/legacy/geolite/
● Support plugins– HTTP, DNS, BGP, SIP/RTP
![Page 14: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/14.jpg)
Adding IP prefix information
MaxMind DB158.38.0.0/16=>UNINETT128.39.0.0/16=>UNINETT
….
158.38.13.0/24=>hials158.38.31.0/24=>hist
128.39.36.0/23=>simula….
MaxMindDB
UNINETTprefix DB
geoipmod
ModifiedMaxMind DB
![Page 15: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/15.jpg)
Appflowag
appflowag
total topapps topas topcountries
IPFIX flow records
IPFIX aggregation records
Export recordsat regular intervals
![Page 16: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/16.jpg)
Appflowag IPFIX records
● Total traffic– flowStartMilliseconds– flowEndMilliseconds– octetDeltaCount– packetDeltaCount– deltaFlowCount– ipVersion
● Top source AS number– flowStartMilliseconds– flowEndMilliseconds– octetTotalCount– packetTotalCount– deltaFlowCount– bgpSourceAsNumber– l7_proto– ipVersion
261, 1378883700000, 1378883999999, 55430087856, 51440429, 792359, 4261, 1378883700000, 1378883999999, 3666166884, 3127366, 73943, 6259, 1378883700000, 1378883999999, 11979801504, 9847245, 29923, 224, 0, 4259, 1378883700000, 1378883999999, 3600748945, 2413758, 9, 42307, 0, 4
![Page 17: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/17.jpg)
Final Appflow architecture
Nprobe
Packets
Appflowag
Flow records
Aggregatedrecords
SQL database Frontend
CollectorsProbes
IPFIX aggregation recordsto anycast address
Nprobe
Nprobe
IP2CountryCustomIP2AS
geoipmod IP2AS
![Page 18: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/18.jpg)
Nprobe and Appflowag performance
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
Packets per second
6500
7000
7500
8000
8500
9000
9500
Flows per second
0
2
4
6
8
10
12
Total CPU usage8 cores for nProbe, 1 for appflowag
0
1
2
3
4
5
6
Gigabit per second
![Page 19: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/19.jpg)
Total processing (19 probes)
0
5000
10000
15000
20000
25000
30000
35000
40000
Flows per second
# o
f flo
ws
0
500000
1000000
1500000
2000000
2500000
Packets per second
# o
f pkt
s
![Page 20: Large scale passive monitoring at 10Gbps on commodity hardware · 0.7 1 1 0 3.3 5 4 0 6.7 10 7 0 10.2 15 13 0 13.9 20 18 0 16.8 25 23 0 20 29.8 31 3.2 64 bytes packet size, two ports,](https://reader035.vdocument.in/reader035/viewer/2022071101/5fdace1c34efb813d83e6102/html5/thumbnails/20.jpg)
Current status and future work
● 30 new monitoring probes being deployed– 19 in full production
● Appflow in full production– Want to improve unknown traffic– Customers wants to add their own prefixes to classify traffic
● Activate nProbe plugins– SIP/RTP, DNS
● Other QoS measurements– Packet reordering, jitter …
● Software will be released– http://software.uninett.no/