applied detection and analysis with flow data - so con 2014

Applied Detection and Analysis Using Network

Flow Data

Chris Sanders & Jason SmithSecurity Onion Conference 2014

Chris Sanders

• Christian & Husband• Kentuckian and South

Carolinian• MS, GSE, et al.• Non-Profit Director• BBQ Pit Master

Jason Smith

• Kentuckian• Car Aficionado• Raspberry Pi Enthusiast• Junkyard Engineer

Applied Network Security Monitoring

“This book should be required reading for all intrusion analysts and those looking to develop a security monitoring program.”

“Written by analysts, for analysts.”

- Amazon Reviewers

Agenda

Flow Data!•Why it’s important•How you can collect it•What you can do with it•Tool that’s can help

“[Why, How] to extend Security Onion with Flow Analysis.“

The NSM Cycle

Evolution of NSM Emphasis

We All Want Full PCAP…

•Collection– Easy to Capture / Filter Stream Data

•Detection– Major Detection Tools are PCAP Oriented

•Analysis– Gives us Who, Where, When, and What

NSM Challenges of the Present

But, It’s not Feasible for Every Goal…

•Collection– Not Scalable for Extended Retention

•Detection– Not Ideal for Hunting / Rapid Pivoting

•Analysis– Not a Great Starting Point

NSM Challenges of the Present

• Often Called Flow / Session / NetFlow• Summary of Network Communications• Aggregated Record of Packets• Gives Us Who, Where, When• Based on the 5-tuple + Timing/Data Stats

Enter Flow Data

Source IP Source Port Dest IP Dest Port Protocol

192.168.5.1 48293 8.8.8.8 53 UDP

• Records are Defined by Unique 5-tuples

• Data is added to the 5-tuple Record until a termination condition is met.

Building Flow Records

• Natural Timeout – End of communication per protocol (ex. RST/FIN)

• Idle Timeout– No data received for 30 seconds

• Active Timeout– Thirty minute max timeout (configurable)

Flow Record Termination Conditions

Full PCAP vs. Flow Data

PCAP Data Flow Data

Level of Context

Full PCAP vs. Flow Data

PCAP Data Flow Data

Storage Requirements

• Most Network Devices Generate it Natively• Collectors are Easy to Setup• Data Footprint is Incredibly Small• Easy for Orgs to Keep Years of Flow Data• Useful for Detection and Analysis

Flow Data Benefits

Collection with Flow Data

• Generation– Routers– Sensors

• Fprobe• YAF

• Multiple Types:– NetFlow (v5,v9)– IPFIX– jFlow– More…

Generating Flow Data

• Popular Platforms– Argus

+ Reliable & Fast Collection- Not Well Supported/Documented

– NFDump+ Easy to Setup and Use- Not in Wide Use

– SiLK+ Exceptional Analysis Tools- More Involved Setup

Collecting Flow Data

• The System for Internet-Level Knowledge• CERT NetSA Team• Two Major Components:

– Packing Suite• Collection and parsing of flow data

– Analysis Suite• Filter, display, sort, count, group, mate, and more

• Excellent Documentation & Community– https://tools.netsa.cert.org/silk/docs.html

SiLK

SiLK Collection Architecture

SiLK – What You Need

Flow Sources− Hardware: Routers, Switches− Software: YAF, fprobe

SiLK Server− Rwflowpack− Will also have SiLK analysis suite installed

Analyst Workstation− Access SiLK server directly − Locally mirrored database

SiLK – Packing Suite Config

rwflowpack – Listens and sorts incoming flows, preparing them for the analysis suite.− --sensor-configuration

Defines listener options Defines ipblocks Defines sensor probes

− --site-config-file Matches sensor probes with a naming convention Defines class and type relationships

− --root-directory Location where all binary flat files are stored

Indexed: Type>Year>Month>Day>Hour

SiLK – Analysis Suite

rwfilter - Filters through data based on conditions. rwcut - Converts flow binary data to a human readable format. rwstats - Generates statistics from flow data rwcount - Summarizes total network traffic over time

SiLK Analysis – rwfilter / rwcut (1)

Display all records from the beginning the current day until the current time:rwfilter --type=all --proto=0-255 --pass=stdout | rwcut

SiLK Analysis – rwfilter / rwcut (2)

Display all records of communication to or from Chinese IP addresses over a specific week to one local CIDR range: rwfilter --type=all --start-date=2014/08/01 --end-date=2014/08/07 --any-address=192.168.1.0/24 --any-cc=cn --pass=stdout | rwcut --fields=stime,sip,dip,sport,dport,type

SiLK Analysis – rwstats (1)

Display statistics for the total amount of bytes transferred by protocol (top 10):rwfilter --type=all --proto=0-255 --pass=stdout | rwstats --top --count=10 --fields=proto --value=bytes

SiLK Analysis – rwstats (2)

Show the top 10 sip,dip pairs for valid conversations (top 10)rwfilter --type=all --proto=0-255 --packets=4, --pass=stdout | rwstats --top --count=10 --fields=sip,dip --value=bytes

SiLK Analysis – rwstats (3)

Show the top 10 outbound destination country codes by records:rwfilter --type=out,outweb --proto=0-255 --pass=stdout | rwstats --top --count=10 --fields=dcc

SiLK Analysis – Zero Access Example (1)

Rwstats to discover potential victimsrwfilter --type=all --dport=16464,16465,16470,16471 --pass=stdout | rwstats --top --fields=sip --value=distinct:dcc --threshold=3

Filter down to only the potential victim machinerwfilter --type=all --start-date=2014/08/02 --end-date=2014/08/03 --saddress=192.168.106.131 --pass=ZA1.rwf

Analyze the data per 10 minute buckets over the course of 24 hours to look for abnormal user data at bizarre times.rwfilter ZA1.rwf --type=all --proto=0-255 --active-time=2014/08/02:00-2014/08/03:00 --pass=stdout | rwcut --bin-size=600

SiLK Analysis – Zero Access Example (2)

• Friendly Intelligence Gathering• Identify Services on the Network• Identify Normal Behaviors of Hosts• Identify “Friends and Family”

– Friends: Who a host communicates with outside the network

– Family: Who a host communicates with inside the network

Collecting Intelligence Data

• Identify SSH Serversrwfilter --type=out --protocol=6 --packets=4- --ack-flag=1 --sport=22 --pass=stdout | rwcut --fields=sip

• Identify Web Serversrwfilter --type=outweb --protocol=6 --packets=4- --ack-flag=1 --sport=80,443,8080 --pass=stdout | rwcut --fields=sip

Identifying Services

Identifying Friends and Family

• Identify Friendsrwfilter --type=out,outweb --saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin --dcidr=192.168.0.0/24 --fail=stdout

• Identify Familyrwfilter --type=out,outweb --saddress=192.168.1.1 --pass=stdout | rwfilter --input-pipe=stdin --dipset=local --fail=stdout

DETECTION with Flow Data

Flow for Detection

FlowPlotter

• Generates Visualizations from Output of Flow Tools

• Useful for Detection-Oriented Statistics• Written in BASH – Flexible/Tweakable• Maintained in GitHub• Browser Independent

FlowPlotter - GeoMaprwfilter ../Sampledata/sample.rw --dcc=us,cn,-- --fail=stdout |

./flowplotter.sh geomap dcc bytes > geomap.html

FlowPlotter – Line Chartrwfilter --type=all --proto=0-255 --pass=stdout | ./flowplotter.sh linechart

600 bytes > linechart.html

FlowPlotter - TreeMaprwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --

proto=0- --type=all --pass=stdout | ./flowplotter.sh treemap dip records > treemap.html

FlowPlotter - PieChartrwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --

proto=0- --type=all --pass=stdout | ./flowplotter.sh piechart dip bytes > piechart.html

FlowPlotter - Barchart/ColumnChartrwfilter ../Sampledata/sample.rw --sport=1025- --dport=1025- --

proto=0- --type=all --pass=stdout | ./flowplotter.sh columnchart dip bytes > columnchart.html

FlowPlotter - BubbleChartrwfilter ../Sampledata/sample.rw --type=all --proto=0-255 --pass=stdout

| ./flowplotter.sh bubblechart sip > bubblechart.html

FlowPlotter - Timelinerwfilter ../Sampledata/sample.rw --proto=0- --dcc=us,-- --fail=stdout |

./flowplotter.sh timeline sip dip > timeline.html

FlowPlotter - Force Directedrwfilter ../Sampledata/sample.rw --scc=kr --proto=0- --type=all --

pass=stdout | ./flowplotter.sh forceopacity sip dip distinct:dport 100 > forcetest.html

FlowPlotter – Asset Discovery

rwfilter ../Sampledata/sample.rw --proto=0- --type=all --pass=stdout | ./flowplotter.sh assetdiscovery > assettest.html

Analysis with Flow Data

Flow in Analysis – PCAP Only

* Based on the First Hour of Analysis

Flow in Analysis – w/ Flow Data

* Based on the First Hour of Analysis

• Be Prepared to Look at a LOT of Line-Based Data

• Very Command Line Oriented• Not Welcoming to Junior-Level Analysts• Hard to Display/Interpret Data Visually

Flow – Barriers to Entry

SiLK Data Output

• Flow Basic Analysis Tool• Graphical Front-End to SiLK• Easy Two-Step Install on SiLK Capable Box

– Install Locally to SiLK Box– Install Remotely and Interact via SSH w/ Keys

• Rapid Pivoting Between Data• Graphing Ability

Conclusion

• Flow Data is Underused and Underrated• Easy to Collect, Enhances Detection & Analysis• Minimal Barriers to Entry

– SiLK (Easy to Install on SO)– Argus (Already Installed on SO)– Bro (Already Installed on SO)

Thanks Folks!• Questions?

– Chris Sanders: chris@chrissanders.org– Jason Smith: jason.smith.webmail@gmail.com

• Blog/Book– http://www.appliednsm.com

• FlowPlotter– http://www.github.com/automayt/FlowPlotter

• FlowBAT – Release in October!