![Page 1: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/1.jpg)
Applied Detection and Analysis Using Network Flow Data
Chris Sanders and Jason Smith
TAP Intel-Based Detection
Mandiant, a FireEye Company
![Page 2: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/2.jpg)
2
Chris Sanders
Christian & Husband Kentuckian and South Carolinian MS, GSE, et al. Non-Profit Director BBQ Pit Master
![Page 3: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/3.jpg)
3
Jason Smith
Kentuckian Car Aficionado Raspberry Pi enthusiast Junkyard Engineer
![Page 4: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/4.jpg)
4
Applied Network Security Monitoring
“This book should be required reading for all intrusion analysts and those looking to develop a security monitoring program.”
“Written by analysts, for analysts.”
- Amazon Reviewers
![Page 5: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/5.jpg)
5
Flow Data! Why it’s important How you can collect it What you can do with it Tools that can help
“Why/How to use Flow Data in NSM/IR”
Agenda
![Page 6: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/6.jpg)
6
The collection, detection, and analysis of network security data.
The goal of NSM is escalation, or to declare that an incident has occurred so that incident response can occur.
Network Security Monitoring
Network Security
Monitoring
Incident Response
![Page 7: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/7.jpg)
7
The NSM Cycle
Collection
DetectionAnalysis
![Page 8: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/8.jpg)
8
Evolution of NSM Emphasis
Past
• Detection Era
Present
• Collection Era
Future
• Analysis Era
![Page 9: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/9.jpg)
9
We All Want Full PCAP…
Collection− Easy to Capture / Filter Stream Data
Detection− Major Detection Tools are PCAP Oriented
Analysis− Gives us Who, Where, When, and What
NSM/IR Challenges of the Present
![Page 10: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/10.jpg)
10
But, It’s not Feasible for Every Goal…
Collection− Not Scalable for Extended Retention
Detection− Not Ideal of Hunting / Rapid Pivoting
Analysis− Not a Great Starting Point
NSM/IR Challenges of the Present
![Page 11: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/11.jpg)
11
Full PCAP vs. Flow Data
PCAP Data Flow Data
![Page 12: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/12.jpg)
12
Often Called Flow / Session / NetFlow Summary of Network Communications Aggregated Record of Packets Gives Us Who, Where, When Based on the 5-tuple + Timing/Data Stats
Flow Data
Source IP Source Port Dest IP Dest Port Protocol
192.168.5.1 48293 8.8.8.8 53 UDP
Start Time End Time Bytes
2014/09/22T00:03:58.756 2014/09/22T00:04:58.756 76
![Page 13: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/13.jpg)
13
sTime| sIP|dPort| dIP|dPort|pro|bytes|
2014/09/22T00:03:58.756| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72|
2014/09/22T00:03:58.999 10.10.120.1| 53| 10.1.188.5| 53| 17| 89|
2014/09/22T00:08:59.012| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72|
2014/09/22T00:08:59.466| 10.10.120.1| 53| 10.1.188.5| 53| 17| 89|
2014/09/22T00:03:58.756| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72|
2014/09/22T00:03:58.999 10.10.120.1| 53| 10.1.188.5| 53| 17| 89|
2014/09/22T00:08:59.012| 10.10.120.1| 53| 10.1.179.5| 53| 17| 72|
2014/09/22T00:08:59.466| 10.10.120.1| 53| 10.1.188.5| 53| 17| 89|
Flow Data Example
![Page 14: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/14.jpg)
14
Records are Defined by Unique 5-tuples
Data is added to the 5-tuple Record until a termination condition is met.
Building Flow Records
![Page 15: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/15.jpg)
15
Natural Timeout − End of communication per protocol (ex. TCP RST/FIN)
Idle Timeout− No data received for 30 seconds
Active Timeout− Thirty minute max timeout (configurable)
Flow Record Termination Conditions
![Page 16: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/16.jpg)
Collection with Flow Data
![Page 17: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/17.jpg)
17
Generation− Routers− Sensors
Fprobe YAF
Multiple Types:− NetFlow (v5,v9)− IPFIX− jFlow− More…
Generating Flow Data
![Page 18: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/18.jpg)
18
Popular Platforms− Argus
+ Reliable + Fast Collection
- Not Well Supported− NFDump
+ Easy to Setup and Use
- Not in Wide Use− SiLK
+ Exceptional Analysis Tools
- More Involved Setup
Collecting Flow Data
![Page 19: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/19.jpg)
19
The System for Internet-Level Knowledge CERT NetSA Team Two Major Components:
− Packing Suite Collection and parsing of flow data
− Analysis Suite Filter, display, sort, count, group, mate, and more
Excellent Documentation & Community− https://tools.netsa.cert.org/silk/docs.html
SiLK
![Page 20: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/20.jpg)
20
SiLK Collection Architecture
![Page 21: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/21.jpg)
21
SiLK – What You Need
Flow Sources− Hardware: Routers, Switches− Software: YAF, fprobe
SiLK Server− Rwflowpack− Will also have SiLK analysis suite installed
Analyst Workstation− Access SiLK server directly − Locally mirrored database
![Page 22: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/22.jpg)
22
SiLK – Analysis Suite
rwfilter - Filters through data based on conditions. rwcut - Converts flow binary data to a human readable
format. rwstats - Generates statistics from flow data rwcount - Summarizes total network traffic over time
![Page 23: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/23.jpg)
23
SiLK Analysis – rwfilter / rwcut (1)
Display all records from the beginning the current day until the current time:rwfilter --type=all --proto=0-255 --pass=stdout | rwcut
![Page 24: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/24.jpg)
24
SiLK Analysis – rwfilter / rwcut (2)
Display all records of communication to or from Chinese IP addresses over a specific week to one local CIDR range: rwfilter --type=all --start-date=2014/08/01 --end-date=2014/08/07 --any-address=192.168.1.0/24 --any-cc=cn --pass=stdout | rwcut --fields=stime,sip,dip,sport,dport,type
![Page 25: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/25.jpg)
25
SiLK Analysis – rwstats (1)
Display statistics for the total amount of bytes transferred by protocol (top 10):
rwfilter --type=all --proto=0-255 --pass=stdout | rwstats --top --count=10 --fields=proto --value=bytes
![Page 26: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/26.jpg)
26
SiLK Analysis – rwstats (2)
Show the top 10 sip,dip pairs for valid conversations (top 10)rwfilter --type=all --proto=0-255 --packets=4, --pass=stdout | rwstats --top --count=10 --fields=sip,dip --value=bytes
![Page 27: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/27.jpg)
27
SiLK Analysis – rwstats (3)
Show the top 10 outbound destination country codes by records:rwfilter --type=out,outweb --proto=0-255 --pass=stdout | rwstats --top --count=10 --fields=dcc
![Page 28: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/28.jpg)
28
SiLK Analysis – Real World Examples
Rwstats to discover potential ZeroAccess victimsrwfilter --type=all --dport=16464,16465,16470,16471 --pass=stdout | rwstats --top --fields=sip --value=distinct:dcc --threshold=3
![Page 29: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/29.jpg)
29
SiLK Analysis – Real World Examples
Discovering outbound data to applications using nonstandard ports.rwfilter Sampledata/sample.rw \--plugin=app-mismatch.so \--type=out,outweb --proto=6 \--sport=1024- \--packets=4- \--flags-initial=S/SURFPACE \--pass=stdout | \rwstats --fields=application,dport --count=100 \--distinct:dport
![Page 30: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/30.jpg)
30
Friendly Intelligence Gathering Identify Services on the Network Identify Normal Behaviors of Hosts Identify “Friends and Family”
− Friends: Who a host communicates with outside the network− Family: Who a host communicates with inside the network
Collecting Intelligence Data
![Page 31: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/31.jpg)
31
Identify SSH Serversrwfilter --type=out --protocol=6 --packets=4- --ack-flag=1 --sport=22 --pass=stdout | rwcut --fields=sip
Identify Web Serversrwfilter --type=outweb --protocol=6 --packets=4- --ack-flag=1 --sport=80,443,8080 --pass=stdout | rwcut --fields=sip
Identifying Services
![Page 32: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/32.jpg)
32
• Identify Friendsrwfilter --type=out,outweb --saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin --dcidr=192.168.0.0/24 --fail=stdout
• Identify Familyrwfilter --type=out,outweb --saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin --dipset=local --fail=stdout
Identifying Friends and Family
![Page 33: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/33.jpg)
Detection with Flow Data
![Page 34: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/34.jpg)
34
Flow for Detection
Signature-Based
Reputation-Based
Behavior-Based
Statistics-Based
![Page 35: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/35.jpg)
35
Generates Visualizations from the Output of Flow Tools Useful for Detection-Oriented Statistics Written in BASH – Flexible/Tweakable/Minimal Free/Open Source - Maintained in GitHub Browser Independent
FlowPlotter
![Page 36: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/36.jpg)
36
FlowPlotter - GeoMap
rwfilter ../Sampledata/sample.rw --dcc=us,cn,-- --fail=stdout | ./flowplotter.sh geomap dcc bytes > geomap.html
![Page 37: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/37.jpg)
37
FlowPlotter - LineChart
rwfilter --type=all --proto=0-255 --pass=stdout | ./flowplotter.sh linechart 600 bytes > linechart.html
![Page 38: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/38.jpg)
38
FlowPlotter - TreeMap
rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --proto=0- --type=out --pass=stdout | ./flowplotter.sh treemap dip records > treemap.html
![Page 39: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/39.jpg)
39
FlowPlotter - Pie/Bar/Column Chart
rwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --proto=0- --type=all --pass=stdout | ./flowplotter.sh piechart dip bytes > piechart.html
![Page 40: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/40.jpg)
40
FlowPlotter - BubbleChart
rwfilter ../Sampledata/sample.rw --type=all --proto=0-255 --pass=stdout | ./flowplotter.sh bubblechart sip > bubblechart.html
![Page 41: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/41.jpg)
41
FlowPlotter - Timeline
rwfilter --proto=0- --type=out --sport=41142 --pass=stdout | ./flowplotter.sh timeline dip sip > timeline.html
![Page 42: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/42.jpg)
42
FlowPlotter - Force Directed
rwfilter ../Sampledata/sample.rw --scc=kr --proto=0- --type=all --pass=stdout | ./flowplotter.sh forceopacity sip dip distinct:dport 100 > forcetest.html
![Page 43: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/43.jpg)
43
FlowPlotter - AssetDiscovery
rwfilter ../Sampledata/sample.rw --proto=0- --type=all --pass=stdout | ./flowplotter.sh assetdiscovery > assettest.html
![Page 44: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/44.jpg)
Analysis with Flow
![Page 45: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/45.jpg)
45
Flow in Analysis – PCAP Only
Analyze / Make Decisions
Reduce Data Set
Find Related Events in Time Range
Scoping Relevant Time Range
Validate Signature TP < 1%
5%
10%
35%
~ 50%
* Based on the First Hour of Analysis
![Page 46: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/46.jpg)
46
Flow in Analysis - Improved
Analyze / Make Decisions
Expand Data Set as Needed
Find Related Events in Time Range
Scoping Relevant Time Range
Validate Signature TP < 1%
5%
10%
5%
~ 80%
* Based on the First Hour of Analysis
![Page 47: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/47.jpg)
47
Be Prepared to Look at a LOT of Line-Based Data Very Command Line Oriented Not Welcoming to Junior-Level Analysts Hard to Display/Interpret Data Visually
Flow – Barriers to Entry
![Page 48: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/48.jpg)
48
SiLK Data Output
![Page 49: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/49.jpg)
49
![Page 50: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/50.jpg)
50
Flow Basic Analysis Tool Graphical Front-End to SiLK Easy Two-Step Install on SiLK Capable Box
− Install Locally to SiLK Box− Install Remotely and Interact via SSH w/ Keys
Rapid Pivoting Between Data Graphing Ability By Analysts, for Analysts
![Page 51: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/51.jpg)
51
Getting Data with FlowBAT (CLI Mode)
![Page 52: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/52.jpg)
52
Getting Data with FlowBAT (Guided Mode)
![Page 53: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/53.jpg)
53
Manipulating FlowBAT Data
![Page 54: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/54.jpg)
54
Pivoting with FlowBAT Data
![Page 55: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/55.jpg)
55
Generating Stats with FlowBAT
![Page 56: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/56.jpg)
56
Generating Stats with FlowBAT
![Page 57: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/57.jpg)
57
Flow Data is Underused and Underrated Easy to Collect, Enhances Detection & Analysis Minimal Barriers to Entry
− SiLK (Easy to Install on SO)− Argus (Already Installed on SO)− Bro (Already Installed on SO)
Conclusion
![Page 58: Applied Detection and Analysis Using Flow Data - MIRCon 2014](https://reader037.vdocument.in/reader037/viewer/2022103013/546b19caaf795902048b4ec8/html5/thumbnails/58.jpg)
58
Questions?− Chris Sanders – [email protected]− Jason Smith – [email protected]
Blog / Book− http://www.appliednsm.com
FlowPlotter− http://www.github.com/automayt/FlowPlotter/
FlowBAT− http://www.flowbat.com
Thanks Folks!