network payload analysis for advanced persistent …...network payload analysis for advanced...
TRANSCRIPT
10000 7/6/2010 1
Network Payload Analysis for Advanced Persistent Threats
Charles Smutz, Lockheed Martin CIRT
2
About Speaker
Name Charles Smutz
Background Sysadmin, Networking, C&A
Current Job Lead Software Developer
Employer Lockheed Martin CIRT
Education Pursuing PhD at GMU
3
Background
• Understanding of APT– Persistent, Organized, Targeted CNE– Typical APT Attack Sequence
• Importance of Threat Focused CND/Security Intelligence
• You’ll have this by end of Summit
4
Topics
• Motivation– Why do network payload analysis
• Suggestions for Capabilities– What data to collect– Importance of Normalized Payload Analysis– Importance of Information Retrieval
• How to implement Capabilities– COTS/FOSS– Build Your Own
5
Why Network Analysis
• Important Data Source– 4n6 and Detection Intertwined
• 4n6 identifies and vets indicators• Detections feed 4n6
– Facilitate Pre-Compromise Detection– Strong Compliment to Host Analysis
• Complete Attack Sequence Analysis
6
Network Analysis Pros/Cons
• Benefits– Passive nature limits impact to network– Omniscience at network tap points– Control over data retention
• Drawbacks– Network forensics requires explicit data
retention– Encryption
7
Net vs. Host Compromise IRPredominately Host Predominately Network
Detection Malware C2 Beacon
Collection Host Logs, Memory Image, Disk Images
Network Logs, Packet Captures
Artifacts Malware, (Deleted) Toolsand Staged Data, Anything in Memory/Swap/Hyberfil• Commands• Passwords• Lateral Movement• Dropped Tools• Exfilled Data
Full Command and Control Decodes
• Commands• Passwords• Lateral Movement• Dropped Tools• Exfilled Data
DamageAssessment
Days/Weeks Hours/Days
8
Beyond FPC
• FPC is expensive, unwieldy
• Strategies for Targeted Data Collection– Network Transaction Logs– Payload Collection– Payload Metadata
• Information Retrieval For Accessibility
9
Network Transaction Logs
• Situational Awareness--Inbound of HTTP Requests– Direct Attacks (SQL injection etc)– Attacker Reconnaissance
• Options:– Sift through FPC– Collect, normalize, centralize all webserver logs– Snarf and reconstruct web activity
• Lots of tools to do this– Bro, Suricata, HTTPry, etc
• What about other protocols?
10
Attacks Moving Up Stack
http://www.sans.org/top-cyber-security-risks/
Document and Multimedia Viewers, Browsers
12
Attacks Moving Up the Stack
From: [email protected]: from open.relay.com([10.10.10.10]) by mx.company.comReceived: from now.bad.com([172.16.1.1]) by mx.relay.comDate: Thu, 17 Jun 2010 12:03:41 -0700 (PDT)Message-Id: <1.1.2.3.5.8@mailer>X-Mailer: SillyMailer v3.14Subject: All your Base are belong to us
Please review attached.
Edward SpoofedSpoofed Inc.301-867‒5309
InfoKey: CreatorInfoValue: Acrobat PDF PrinterInfoKey: AuthorInfoValue: TK421InfoKey: ProducerInfoKey: ModDateInfoValue: D:20100616+08'00'PdfID1: 8d23f593e67be992ff3470dPdfID0: 798f9d8e3966ac586a61dc0
for(fqchp=0;fqchp<inxnh;fqchp++) {dnysj[fqchp]=dtkrx + hjnoa;}
if(ingmh){hsbsd();hsbsd();try {this.media.newPlayer(null);} catch(e) {}hsbsd();}
<Obfuscated Embedded Malware>
13
Attacks Moving up Stack
Layer Protocol Badness
EmbeddedObject
Application
Transport
Internet
Link
Email from legitimate email relay with Trojan Document Attachment
Ethernet -
IP -
TCP -
SMTP/MIME Spoofed Sender, SocialEngineering
PDF Exploit/Social Engineering,Malware
15
Indicators Moving Up the Stack
From: [email protected]: from open.relay.com([10.10.10.10]) by mx.company.comReceived: from now.bad.com([172.16.1.1]) by mx.relay.comDate: Thu, 17 Jun 2010 12:03:41 -0700 (PDT)Message-Id: <1.1.2.3.5.8@mailer>X-Mailer: SillyMailer v3.14Subject: All your Base are belong to us
Please review attached.
Edward SpoofedSpoofed Inc.301-867‒5309
InfoKey: CreatorInfoValue: Acrobat PDF PrinterInfoKey: AuthorInfoValue: TK421InfoKey: ProducerInfoKey: ModDateInfoValue: D:20100616+08'00'PdfID1: 8d23f593e67be992ff3470dPdfID0: 798f9d8e3966ac586a61dc0
for(fqchp=0;fqchp<inxnh;fqchp++) {dnysj[fqchp]=dtkrx + hjnoa;}
if(ingmh){hsbsd();hsbsd();try {this.media.newPlayer(null);} catch(e) {}hsbsd();}
<Obfuscated Embedded Malware>
12:03:31.165239 tcp 10.10.10.10.59170 -> 192.168.0.10.25 276 29770 FIN
17
Email Data Collection Options
• Basic Email Transaction Data• Network Flow Data• Full Packet Capture• Normalized Emails
– Reassembled, Decoded, Indexed• Extended Email Metadata
– Headers: Subject, X-Mailer, Received– MIME Metadata: Names, Size, md5– Links
• Attachments (specific type?)• Attachment Metadata: Author, Creator, Dates
19
Tiered Collection
Data RetentionLength
Size / Day Total Size
FPC (entire network) 1 week 1 TB 7 TB
Network Flow (entire network) 1 year 4 GB 1.5 TBStandard Mail Logs 2 year 50 MB 36 GB
Normalized, Indexed Emails 6 weeks 20 GB 800 GBExtended Email Metadata 6 months 500 MB 100 GB
Attachment Metadata 6 months 100 MB 20 GB
20
Accessibility Is Critical
• Rapid accessibility is critical:– Historical Detections– Identifying and vetting indicators
• Time to research an indicator matters– 1s, 1 minute, 1 hour, 1 day?
The faster you can research activity over large spans of time, the faster you’ll build threat intelligence
21
From: [email protected]: from open.relay1.com ([10.10.10.10]) by mx.company.comReceived: from now.bad.com ([172.16.1.1]) by mx.relay.comDate: Mon, 28 Dec 2009 5:48:02 +0800Message-Id: <1.1.2.3.5.8@mailer>X-Mailer: SillyMailer v3.14
<Malware 1.3>
From: [email protected]: from mx.openrelay2.com ([10.20.30.40]) by mx.company.comReceived: from now.bad.com ([192.168.2.2]) by mail.openrelay2.comDate: Mon, 5 Mar 2010 13:35:28 -0700 (PDT)Message-Id: <1.1.2.3.6.9@mailer>X-Mailer: SillyMailer v3.14
<Malware 2.0>
From: [email protected]: from relay.all.com ([10.70.50.60]) by mx.company.comReceived: from now.bad.com ([172.16.1.1]) by mx.relay.comDate: Thu, 17 Jun 2010 12:03:41 -0700 (PDT)Message-Id: <1.1.2.3.7.2@mailer>X-Mailer: SillyMailer v3.14
<Malware 2.01>
22
Ultra Light Weight Indexing
• Rapidly Search Key Indicator Types– IP addresses, Domains, etc
• Low Resolution– Log Type: proxy, email, etc– Time: ~Day– Per Device: proxy1, proxy2, proxy3
• Huge Scope– Time: indefinite retention– Data Sources: All
• Performance– Fast, << 1s response times
23
Ultra Light Weight Indexing
Data Type Source Date Indicatoremail-metadata mx1 2009-12-28 172.16.1.1inbound-http sensor1 2010-03-04 172.16.1.1email-metadata mx2 2010-06-17 172.16.1.1
Example search for 172.16.1.1:
24
Implementing Payload Analysis Tools
• Passive Collection:– Adapt an FPC
• Tail collection, filter normalize, extract – Adapt an IDS
• Filter, normalize, extract, archive• Inline Collection
– Milter, ICAP, etc
• Differences probably nuances, End goal is the same
25
Payload Analysis Issues
• Issues to be addressed:– Latency– Computational Expense– Implementing Payload Specific Capabilities
26
Payload Analysis: Latency
• IDS/IPS bound by real time• FPC provides on-demand data/processing
(arbitrarily long)• High Latency Analysis to be preformed (lookups)• Payload analysis for 4n6 usually should be
somewhere in between– Usually no benefit to be quicker than minute– For some applications slower than hour can
slow down response– Often daily processing makes sense
27
Payload Analysis: Complexity
• Expensive Tasks– Decoding, decompression, etc– Parsing, tokenizing, metadata extraction– Normalized archival (buffer copies)– Payload Identification– Any inherently computationally expensive
things• Statistical analysis• Compression• Etc
28
Latency and Complexity
• Heavy Buffering– 1 Gpbs * 60s = 7.5 GB RAM (dirt cheap)
• True Parallelism– Load balancing needs to move up stack also
• Example later
29
Implementing Payload Specific Capabilities
• Use existing network capabilities• Protocol Parsers
– HTTP::Parser, Mime::Parser, etc• Use payload capabilities• Payload Analyzers
– pdftk, pdf-parser, Officecat, etc• Use your in-house tools on extracted payloads
– Build network tools that work on objects (Abstraction)
30
Near Real Time IDS Platforms
• vortex (Lockheed Martin)– http://sourceforge.net/projects/vortex-ids/– Abstracts capture and TCP stream reassembly,
simple method for multithreading• snort-nrt (Sourcefire VRT)
– http://labs.snort.org/nrt/– Commitment to payload analysis
• Ruminate (George Mason University)– http://mason.gmu.edu/~csmutz/ruminate/– Focus on efficiency, scalability, completeness
of parsing
31
Vortex Overview
VortexStream Management, Flow Control
LibnidsTCP Stream Reassembly
LibpcapPacket Capture/Filtering
Cap
ture
d N
etw
ork
Traf
fic
File SystemSt
ream
D
ata
Analyzer ProgramReads Metadata, Loads Stream Data, Analyzes, optionally Purges
Stream Data
Stre
am M
etad
ata
(STD
OU
T)
32
Vortex Multithreaded
Vortex
Cap
ture
d N
etw
ork
Traf
fic
File SystemStre
am
Dat
a
Analyzer Program
Stre
am M
etad
ata
(STD
OU
T)
Xpip
esLo
ad B
alan
cing
Analyzer Program
Analyzer Program
Analyzer Program
33
Conclusions
• Network Data is important source for 4n6• Strategies for Network Data Collection
– Conventional (netflow, logs, FPC)– Targeted (playloads, payload metadata)
• Importance of data accessibility– Normalization– Search and Retrieval
• Ideas on Implementation
36
APT Attack Sequence
Reconnaissance Weaponization Delivery Exploit Installation
Pre-Compromise
Post-Compromise
Reconnaissance Initial Intrusion
Establish Backdoor
Obtain User Credentials
Install Various Utilities
Priv. Escalation, Lateral Move.,
Data Exfil.
Maintain Persistance
Command & Control Actions on Intent